News Article Recommender

News Article Recommender

Often, we run across a news article that sparks our interest in a particular topic or theme, leaving us curious, or even hungry, to learn more. We want to build on that momentum by reading more articles on the topic right away. Article recommendation apps make this easier by surfacing similar articles to the one we are reading, so we can expand our understanding as much as we want.

Our News Article Recommender app is based on the concept of text embeddings, which capture the meaning of a piece of text beyond simple keyword matching. It compares the embeddings similarity between a selected article to all other available articles. The app refines the list of candidate articles by using text classification to only recommend articles within the same category (e.g., sports or business). It then extracts a list of tags from each recommended article and displays them alongside the article to help the reader scan and choose the next article to read.

All of this is done via three Cohere API endpoints stacked together: Embed, Classify, and Generate. The steps to build the News Article Recommender are:

Step 1: Get a list of articles
Step 2: Embed articles
Step 3: Create article selector
Step 4: Find the most similar articles
Step 5: Classify each article’s category
Step 6: Extract tags from these articles
Step 7: Put everything together

Read on for more details on each of these steps.

Building an Article Recommender App

Step 1: Get a List of Articles

First, we’ll need to load the articles. In this example, we’re using the BBC news article dataset [source], which consists of articles from a few categories: business, politics, tech, entertainment, and sports, and we’re focusing on a subset of this dataset containing 100 articles.

# Load the dataset to a dataframe
def load_df(csv_file):
    return pd.read_csv(csv_file, delimiter=',')

df = load_df('bbc_news_test.csv')
articles = df["news"].tolist()
titles = df["title"].tolist()

Step 2: Embed Articles

Next, we’ll turn each article's text into embeddings. An embedding is a list of numbers that a model uses to represent a piece of text, capturing its context and meaning. We do this by calling the Cohere Embed endpoint, which takes in texts as input and returns embeddings as output.

# Get text embeddings via the Embed endpoint
def embed_text(texts):
    embeddings = co.embed(
                texts = texts)
    embeddings = np.array(embeddings.embeddings)
    return embeddings

Step 3: Create Article Selector

We’ll then create a Streamlit selectbox where a reader can select the article that he or she is currently reading (let's call this the target).

# Drop down selectbox
col1, col2 = st.columns(2)
with col1:
   current_id = st.selectbox(
   "Select an Article",
   format_func=lambda x: titles[x])

Step 4: Find the Most Similar Articles

Next, we’ll create a function to find other articles with the most similar embeddings (let's call these candidates) using cosine similarity.

Cosine similarity is a metric that measures how similar two sequences of numbers are (embeddings in our case), and we compute it for each target-candidate pair.

# Calculate cosine similarity between the target and candidate articles
def get_similarity(target,candidates):
   # Turn list into array
   candidates = np.array(candidates)
   target = np.expand_dims(np.array(target),axis=0)
   # Calculate cosine similarity
   similarity_scores = cosine_similarity(target,candidates)
   similarity_scores = np.squeeze(similarity_scores).tolist()
   # Sort by descending order in similarity
   similarity_scores = list(enumerate(similarity_scores))
   similarity_scores = sorted(similarity_scores, key=lambda x:x[1], reverse=True)
   # Return similarity scores
   return similarity_scores
# Get the similarity between the target and candidate articles
embs = rec.embed_text(articles)
current_emb = embs[current_id]
similarity_scores = rec.get_similarity(current_emb,embs)

Step 5: Classify Each Article’s Category

In the context of news articles, it’s important to note that two articles may have similar meanings but can fall under totally different categories. For example, an article about a player’s anger over a racism fine (Sports) can appear to be similar to an article about a politician’s disagreement over an apology (Politics) because both articles revolve around confrontations and the clash of individuals.

In our case, we don’t want to recommend articles from a different category even if they appear to be similar to the current article. For this, we need to build a news category classifier that will only select articles from the same category.

We’ll use the Cohere Classify endpoint to build a news category classifier that classifies articles into five classes: Business, Politics, Tech, Entertainment, and Sports. For this app, we’ll train a custom model using another subset of the BBC news article. Find more information in the training a representation model documentation.

def classify_text(input_text):
   text = truncate_text(input_text)
   classifications = co.classify(
   return classifications.classifications[0].prediction
current_class = rec.classify_text(articles[current_id])

Step 6: Extract Tags from These Articles

Next, we’ll want to enrich the recommended articles with more information to help users scan the list for key information and discover content. The outcome: each recommended article will be displayed with a list of tags containing the key topics mentioned in the article.

We’ll build this tags extractor via the Cohere Generate endpoint. First, we create a prompt with a few examples of text and its tags. Then we call the Generate endpoint using the following prompt, appending each recommended article to get the recommended tags.

Refer to the prompt engineering documentation if you’d like to know more about creating prompts for the Generate endpoint.

# Create the prompt for tags extraction
def create_prompt(text):
   return f"""Given a news article, this program returns the list tags containing keywords of that article.
Article: japanese banking battle at an end japan s sumitomo mitsui financial has withdrawn its takeover offer for rival bank ufj holdings  enabling the latter to merge with mitsubishi tokyo.  sumitomo bosses told counterparts at ufj of its decision on friday  clearing the way for it to conclude a 3 trillion
Tags: sumitomo mitsui financial, ufj holdings, mitsubishi tokyo, japanese banking
Article: france starts digital terrestrial france has become the last big european country to launch a digital terrestrial tv (dtt) service.  initially  more than a third of the population will be able to receive 14 free-to-air channels. despite the long wait for a french dtt roll-out
Tags: france, digital terrestrial
Article: apple laptop is  greatest gadget  the apple powerbook 100 has been chosen as the greatest gadget of all time  by us magazine mobile pc.  the 1991 laptop was chosen because it was one of the first  lightweight  portable computers and helped define the layout of all future notebook pcs.
Tags: apple, apple powerbook 100, laptop
Article: {text}
# Get extractions via the Generate endpoint
def extract_tags(complete_prompt):
   # Truncate the complete prompt if too long
   token_check = co.tokenize(text=complete_prompt)
   if len(token_check.tokens) > 2000:
       complete_prompt = co.detokenize(token_check.tokens[:2000]).text
   prediction = co.generate(
   tags_raw =  prediction.generations[0].text
   if "\n" in tags_raw:
       tags_clean =".+?(?=\\n)",tags_raw).group()
       tags_clean = tags_raw
   if tags_clean:
       tags = tags_clean.split(",")
       tags = list(dict.fromkeys(tags)) # remove duplicates
       tags = [tag.strip() for tag in tags] # remove empty spaces
       tags = [tag for tag in tags if tag] # remove empty tags
       tags = [tag for tag in tags if len(tag) > 3] # remove short tags
       tags = [f"`{tag}`" for tag in tags] # format tag string
       tags = ",".join(tags)
       tags = "None"
   return tags

tags = rec.extract_tags(complete_prompt)

Step 7: Put Everything Together

Finally, we’ll tie together all the steps that we’ve completed so far: embedding, finding the most similar articles, news category classification, and tags extraction.

We then render the Streamlit components to display the current and recommended articles.

# Current article
with col3:
   st.markdown("***You are currently reading...***")
   current_class = rec.classify_text(articles[current_id])
   st.markdown(f"###### {current_class.capitalize()}")
   current_article = articles[current_id]
# Recommended article
with col4:
   st.markdown("***You might also like...***")
   SHOW_TOP = 3
   count = 0
   for candidate_id,_ in similarity_scores:
       candidate_article = articles[candidate_id]
       candidate_title = titles[candidate_id]
       candidate_class = rec.classify_text(candidate_article)
       if candidate_class ==  current_class and candidate_id != current_id:
           # Show title
           st.markdown(f"###### {candidate_class.capitalize()}")
           # Show article
           expander = st.expander("Read article")
           # Show tags
           complete_prompt = rec.create_prompt(candidate_article)
           tags = rec.extract_tags(complete_prompt)
           count += 1
       if count == SHOW_TOP:


And that concludes the process of creating our News Article Recommender app, built via three Cohere API endpoints stacked together: Embed, Classify, and Generate. To get started building your own version, create a free Cohere account.