Fueling Generative Content with Keyword Research

Back to Cookbooks Open in GitHub

Generative models have proven extremely useful in content idea generation. But they don’t take into account user search demand and trends. In this notebook, let’s see how we can solve that by adding keyword research into the equation.

Read the accompanying blog post here.

PYTHON

1 ! pip install cohere -q

PYTHON

1 import cohere
2 import numpy as np
3 import pandas as pd
4 from sklearn.cluster import KMeans
5 
6 import cohere
7 co = cohere.Client("COHERE_API_KEY") # Get your API key: https://dashboard.cohere.com/api-keys

PYTHON

1 #@title Enable text wrapping in Google Colab
2 
3 from IPython.display import HTML, display
4 
5 def set_css():
6   display(HTML('''
7 
8   '''))
9 get_ipython().events.register('pre_run_cell', set_css)

First, we need to get a supply of high-traffic keywords for a given topic. We can get this via keyword research tools, of which are many available. We’ll use Google Keyword Planner, which is free to use.

PYTHON

1 import wget
2 wget.download("https://raw.githubusercontent.com/cohere-ai/cohere-developer-experience/main/notebooks/data/remote_teams.csv", "remote_teams.csv")

'remote_teams.csv'

PYTHON

1 df = pd.read_csv('remote_teams.csv')
2 df.columns = ["keyword","volume"]
3 df.head()

	keyword	volume
0	managing remote teams	1000
1	remote teams	390
2	collaboration tools for remote teams	320
3	online games for remote teams	320
4	how to manage remote teams	260

We now have a list of keywords, but this list is still raw. For example, “managing remote teams” is the top-ranking keyword in this list. But at the same time, there are many similar keywords further down in the list, such as “how to effectively manage remote teams.”

We can do that by clustering them into topics. For this, we’ll leverage Cohere’s Embed endpoint and scikit-learn.

Embed the Keywords with Cohere Embed

The Cohere Embed endpoint turns a text input into a text embedding.

PYTHON

1 def embed_text(texts):
2   output = co.embed(
3                 texts=texts,
4                 model='embed-english-v3.0',
5                 input_type="search_document",
6                 )
7   return output.embeddings
8 
9 embeds = np.array(embed_text(df['keyword'].tolist()))

Cluster the Keywords into Topics with scikit-learn

We then use these embeddings to cluster the keywords. A common term used for this exercise is “topic modeling.” Here, we can leverage scikit-learn’s KMeans module, a machine learning algorithm for clustering.

PYTHON

1 NUM_TOPICS = 4
2 kmeans = KMeans(n_clusters=NUM_TOPICS, random_state=21, n_init="auto").fit(embeds)
3 df['topic'] = list(kmeans.labels_)
4 df.head()

	keyword	volume	topic
0	managing remote teams	1000	0
1	remote teams	390	1
2	collaboration tools for remote teams	320	1
3	online games for remote teams	320	3
4	how to manage remote teams	260	0

Generate Topic Names with Cohere Chat

We use the Chat to generate a topic name for that cluster.

PYTHON

1 topic_keywords_dict = {topic: list(set(group['keyword'])) for topic, group in df.groupby('topic')}

PYTHON

1 def generate_topic_name(keywords):
2     # Construct the prompt
3     prompt = f"""Generate a concise topic name that best represents these keywords.\
4 Provide just the topic name and not any additional details.
5 
6 Keywords: {', '.join(keywords)}"""
7 
8     # Call the Cohere API
9     response = co.chat(
10         model='command-r',  # Choose the model size
11         message=prompt,
12         preamble="")
13 
14     # Return the generated text
15     return response.text

PYTHON

1 topic_name_mapping = {topic: generate_topic_name(keywords) for topic, keywords in topic_keywords_dict.items()}
2 
3 df['topic_name'] = df['topic'].map(topic_name_mapping)
4 
5 df.head()

	keyword	volume	topic	topic_name
0	managing remote teams	1000	0	Remote Team Management
1	remote teams	390	1	Remote Team Tools and Tips
2	collaboration tools for remote teams	320	1	Remote Team Tools and Tips
3	online games for remote teams	320	3	Remote Team Fun
4	how to manage remote teams	260	0	Remote Team Management

PYTHON

1 for topic, name in topic_name_mapping.items():
2     print(f"Topic {topic}: {name}")

Topic 0: Remote Team Management
Topic 1: Remote Team Tools and Tips
Topic 2: Remote Team Resources
Topic 3: Remote Team Fun

Now that we have the keywords nicely grouped into topics, we can proceed to generate the content ideas.

Take the Top Keywords from Each Topic

Here we can implement a filter to take just the top N keywords from each topic, sorted by the search volume. In our case, we use 10.

PYTHON

1 TOP_N = 10
2 
3 top_keywords = (df.groupby('topic')
4                         .apply(lambda x: x.nlargest(TOP_N, 'volume'))
5                         .reset_index(drop=True))
6 
7 
8 content_by_topic = {}
9 for topic, group in top_keywords.groupby('topic'):
10     keywords = ', '.join(list(group['keyword']))
11     topic2name = topic2name = dict(df.groupby('topic')['topic_name'].first())
12     topic_name = topic2name[topic]
13     content_by_topic[topic] = {'topic_name': topic_name, 'keywords': keywords}

PYTHON

1 content_by_topic

{0: {'topic_name': 'Remote Team Management',
  'keywords': 'managing remote teams, how to manage remote teams, leading remote teams, managing remote teams best practices, remote teams best practices, best practices for managing remote teams, manage remote teams, building culture in remote teams, culture building for remote teams, managing remote teams training'},
 1: {'topic_name': 'Remote Team Tools and Tips',
  'keywords': 'remote teams, collaboration tools for remote teams, team building for remote teams, scrum remote teams, tools for remote teams, zapier remote teams, working agreements for remote teams, working with remote teams, free collaboration tools for remote teams, free retrospective tools for remote teams'},
 2: {'topic_name': 'Remote Team Resources',
  'keywords': 'best collaboration tools for remote teams, slack best practices for remote teams, best communication tools for remote teams, best tools for remote teams, always on video for remote teams, best apps for remote teams, best free collaboration tools for remote teams, best games for remote teams, best gifts for remote teams, best ice breaker questions for remote teams'},
 3: {'topic_name': 'Remote Team Fun',
  'keywords': 'online games for remote teams, team building activities for remote teams, games for remote teams, retrospective ideas for remote teams, team building ideas for remote teams, fun retrospective ideas for remote teams, retro ideas for remote teams, team building exercises for remote teams, trust building exercises for remote teams, activities for remote teams'}}

Create a Prompt with These Keywords

Next, we use the Chat endpoint to produce the content ideas. The prompt we’ll use is as follows

PYTHON

1 def generate_blog_ideas(keywords):
2   prompt = f"""{keywords}\n\nThe above is a list of high-traffic keywords obtained from a keyword research tool.
3 Suggest three blog post ideas that are highly relevant to these keywords.
4 For each idea, write a one paragraph abstract about the topic.
5 Use this format:
6 Blog title: <text>
7 Abstract: <text>"""
8 
9   response = co.chat(
10     model='command-r',
11     message = prompt)
12   return response.text

Generate Content Ideas

Next, we generate the blog post ideas. It takes in a string of keywords, calls the Chat endpoint, and returns the generated text.

PYTHON

1 for key,value in content_by_topic.items():
2   value['ideas'] = generate_blog_ideas(value['keywords'])
3 
4 
5 for key,value in content_by_topic.items():
6   print(f"Topic Name: {value['topic_name']}\n")
7   print(f"Top Keywords: {value['keywords']}\n")
8   print(f"Blog Post Ideas: {value['ideas']}\n")
9   print("-"*50)

Topic Name: Remote Team Management
Top Keywords: managing remote teams, how to manage remote teams, leading remote teams, managing remote teams best practices, remote teams best practices, best practices for managing remote teams, manage remote teams, building culture in remote teams, culture building for remote teams, managing remote teams training
Blog Post Ideas: Here are three blog post ideas:
1. Blog title: "Leading Remote Teams: Strategies for Effective Management"
   Abstract: Effective management of remote teams is crucial for success, but it comes with unique challenges. This blog will explore practical strategies for leading dispersed employees, focusing on building a cohesive and productive virtual workforce. It will cover topics such as establishing clear communication protocols, fostering a collaborative environment, and the importance of trusting and empowering your remote employees for enhanced performance.
2. Blog title: "Remote Teams' Best Practices: Creating a Vibrant and Engaging Culture"
   Abstract: Building a rich culture in a remote team setting is essential for employee engagement and retention. The blog will delve into creative ways to foster a sense of community and connection among team members who may be scattered across the globe. It will offer practical tips on creating virtual rituals, fostering open communication, and harnessing the power of technology for cultural development, ensuring remote employees feel valued and engaged.
3. Blog title: "Managing Remote Teams: A Comprehensive Guide to Training and Development"
   Abstract: Training and developing remote teams present specific challenges and opportunities. This comprehensive guide will arm managers with techniques to enhance their remote team's skills and knowledge. It will explore the latest tools and methodologies for remote training, including virtual workshops, e-learning platforms, and performance coaching. Additionally, the blog will discuss the significance of ongoing development and how to create an environment that encourages self-improvement and learning.
Each of these topics explores a specific aspect of managing remote teams, providing valuable insights and practical guidance for leaders and managers in the evolving remote work landscape.
--------------------------------------------------
Topic Name: Remote Team Tools and Tips
Top Keywords: remote teams, collaboration tools for remote teams, team building for remote teams, scrum remote teams, tools for remote teams, zapier remote teams, working agreements for remote teams, working with remote teams, free collaboration tools for remote teams, free retrospective tools for remote teams
Blog Post Ideas: 1. Blog title: "The Ultimate Guide to Building Effective Remote Teams"
   Abstract: Building a cohesive and productive remote team can be challenging. This blog will serve as a comprehensive guide, offering practical tips and insights on how to create a united and successful virtual workforce. It will cover essential topics such as building a strong team culture, utilizing collaboration tools, and fostering effective communication strategies, ensuring remote teams can thrive and achieve their full potential.
2. Blog title: "The Best Collaboration Tools for Remote Teams: A Comprehensive Review"
   Abstract: With the rapid rise of remote work, collaboration tools have become essential for teams' productivity and efficiency. This blog aims to review and compare the most popular collaboration tools, providing an in-depth analysis of their features, ease of use, and benefits. It will offer insights into choosing the right tools for remote collaboration, helping teams streamline their workflows and enhance their overall performance.
3. Blog title: "Remote Retrospective: A Guide to Reflect and Grow as a Remote Team"
   Abstract: Conducting effective retrospectives is crucial for remote teams to reflect on their experiences, learn from the past, and chart a course for the future. This blog will focus on remote retrospectives, exploring different formats, techniques, and free tools that teams can use to foster continuous improvement. It will also provide tips on creating a safe and inclusive environment, encouraging honest feedback and productive discussions.
--------------------------------------------------
Topic Name: Remote Team Resources
Top Keywords: best collaboration tools for remote teams, slack best practices for remote teams, best communication tools for remote teams, best tools for remote teams, always on video for remote teams, best apps for remote teams, best free collaboration tools for remote teams, best games for remote teams, best gifts for remote teams, best ice breaker questions for remote teams
Blog Post Ideas: 1. Blog title: "The Ultimate Guide to Remote Team Collaboration Tools"
   Abstract: With the rise of remote work, choosing the right collaboration tools can be crucial to a team's success and productivity. This blog aims to be an comprehensive guide, outlining the various types of tools available, from communication platforms like Slack to project management software and online collaboration tools. It will offer best practices and guidelines for selecting and utilizing these tools, ensuring remote teams can work seamlessly together and maximize their output.
2. Blog title: "Remote Team Management: Tips for Leading a Successful Virtual Workforce"
   Abstract: Managing a remote team comes with its own set of challenges. This blog will provide an in-depth exploration of best practices for leading and motivating virtual teams. Covering topics such as effective communication strategies, performance evaluation, and maintaining a cohesive team culture, it will offer practical tips for managers and leaders to ensure their remote teams are engaged, productive, and well-managed.
3. Blog title: "The Fun Side of Remote Work: Games, Icebreakers, and Team Building Activities"
   Abstract: Remote work can be isolating, and this blog aims to provide some fun and creative solutions. It will offer a comprehensive guide to the best online games, icebreaker questions, and virtual team building activities that remote teams can use to connect and bond. From virtual escape rooms to interactive games and thought-provoking icebreakers, these ideas will help enhance team spirit, foster collaboration, and create a enjoyable remote work experience.
--------------------------------------------------
Topic Name: Remote Team Fun
Top Keywords: online games for remote teams, team building activities for remote teams, games for remote teams, retrospective ideas for remote teams, team building ideas for remote teams, fun retrospective ideas for remote teams, retro ideas for remote teams, team building exercises for remote teams, trust building exercises for remote teams, activities for remote teams
Blog Post Ideas: 1. Blog title: "The Great Remote Retro: Fun Games and Activities for Your Team"
   Abstract: Remote work can make team building challenging. This blog post will be a fun guide to hosting interactive retro games and activities that bring your remote team together. From online escape rooms to virtual scavenger hunts, we'll explore the best ways to engage and unite your team, fostering collaboration and camaraderie. Virtual icebreakers and retrospective ideas will also be included to make your remote meetings more interactive and enjoyable.
2. Blog title: "Trust Falls: Building Trust Among Remote Teams"
   Abstract: Trust is the foundation of every successful team, but how do you build it when everyone is scattered across different locations? This blog will focus on trust-building exercises and activities designed specifically for remote teams. From virtual trust falls to transparent communication practices, we'll discover innovative ways to strengthen team bonds and foster a culture of trust. You'll learn how to create an environment where your remote team can thrive and collaborate effectively.
3. Blog title: "Game Night for Remote Teams: A Guide to Online Games and Activities"
   Abstract: Miss the old office game nights? This blog will bring the fun back with a guide to hosting online game nights and activities that are perfect for remote teams. From trivia games to virtual board games and even remote-friendly outdoor adventures, we'll keep your team engaged and entertained. With tips on setting up online tournaments and ideas for encouraging participation, your virtual game nights will be the highlight of your team's week. Keep your remote team spirit high!
--------------------------------------------------

1	import cohere
2	import numpy as np
3	import pandas as pd
4	from sklearn.cluster import KMeans
5
6	import cohere
7	co = cohere.Client("COHERE_API_KEY") # Get your API key: https://dashboard.cohere.com/api-keys

1	#@title Enable text wrapping in Google Colab
2
3	from IPython.display import HTML, display
4
5	def set_css():
6	display(HTML('''
7
8	'''))
9	get_ipython().events.register('pre_run_cell', set_css)

1	import wget
2	wget.download("https://raw.githubusercontent.com/cohere-ai/cohere-developer-experience/main/notebooks/data/remote_teams.csv", "remote_teams.csv")

1	df = pd.read_csv('remote_teams.csv')
2	df.columns = ["keyword","volume"]
3	df.head()

1	def embed_text(texts):
2	output = co.embed(
3	texts=texts,
4	model='embed-english-v3.0',
5	input_type="search_document",
6	)
7	return output.embeddings
8
9	embeds = np.array(embed_text(df['keyword'].tolist()))

1	NUM_TOPICS = 4
2	kmeans = KMeans(n_clusters=NUM_TOPICS, random_state=21, n_init="auto").fit(embeds)
3	df['topic'] = list(kmeans.labels_)
4	df.head()

1	def generate_topic_name(keywords):
2	# Construct the prompt
3	prompt = f"""Generate a concise topic name that best represents these keywords.\
4	Provide just the topic name and not any additional details.
5
6	Keywords: {', '.join(keywords)}"""
7
8	# Call the Cohere API
9	response = co.chat(
10	model='command-r', # Choose the model size
11	message=prompt,
12	preamble="")
13
14	# Return the generated text
15	return response.text

1	topic_name_mapping = {topic: generate_topic_name(keywords) for topic, keywords in topic_keywords_dict.items()}
2
3	df['topic_name'] = df['topic'].map(topic_name_mapping)
4
5	df.head()

1	for topic, name in topic_name_mapping.items():
2	print(f"Topic {topic}: {name}")

1	TOP_N = 10
2
3	top_keywords = (df.groupby('topic')
4	.apply(lambda x: x.nlargest(TOP_N, 'volume'))
5	.reset_index(drop=True))
6
7
8	content_by_topic = {}
9	for topic, group in top_keywords.groupby('topic'):
10	keywords = ', '.join(list(group['keyword']))
11	topic2name = topic2name = dict(df.groupby('topic')['topic_name'].first())
12	topic_name = topic2name[topic]
13	content_by_topic[topic] = {'topic_name': topic_name, 'keywords': keywords}

1	def generate_blog_ideas(keywords):
2	prompt = f"""{keywords}\n\nThe above is a list of high-traffic keywords obtained from a keyword research tool.
3	Suggest three blog post ideas that are highly relevant to these keywords.
4	For each idea, write a one paragraph abstract about the topic.
5	Use this format:
6	Blog title: <text>
7	Abstract: <text>"""
8
9	response = co.chat(
10	model='command-r',
11	message = prompt)
12	return response.text

1	for key,value in content_by_topic.items():
2	value['ideas'] = generate_blog_ideas(value['keywords'])
3
4
5	for key,value in content_by_topic.items():
6	print(f"Topic Name: {value['topic_name']}\n")
7	print(f"Top Keywords: {value['keywords']}\n")
8	print(f"Blog Post Ideas: {value['ideas']}\n")
9	print("-"*50)