Nucleus Sampling (top_p)

Nucleus sampling (top_p) is a parameter used in the generation of text that controls the diversity of the generated responses. It is an alternative sampling technique to temperature, and OpenAI recommends altering either top_p or temperature, not both. Top-p sampling is a probabilistic approach where the model considers the cumulative probability of the next word and selects from the smallest set of words whose cumulative probability exceeds a threshold. The threshold is determined by the value of top_p, which can range from 0 to 1. A lower value of top_p means that the model will only consider the most likely words, resulting in more predictable and repetitive outputs. A higher value of top_p means that the model will consider a larger set of words, resulting in more diverse and creative outputs.

How Nucleus sampling affect your translation

Nucleus sampling works by selecting the next word from the smallest set of words whose cumulative probability exceeds a certain threshold. For example, if the threshold is 0.9, then the model will only consider the words that have a combined probability of at least 0.9. This way, the model can avoid choosing words that are very unlikely and generate more coherent and relevant texts .

The range of top_p is from 0 to 1, where 0 means no sampling (the model always chooses the most probable word) and 1 means full sampling (the model considers all possible words). A lower value of top_p will result in more predictable and repetitive texts, while a higher value will result in more diverse and unpredictable texts.

For translation requests, the value of top_p should depend on the desired trade-off between accuracy and fluency. A lower value of top_p may produce more accurate translations, but they may also sound unnatural or awkward. A higher value of top_p may produce more fluent translations, but they may also introduce errors or hallucinations. Therefore, it is advisable to experiment with different values of top_p and evaluate the quality of the translations using metrics such as BLEU or human ratings.

Examples of different value

Here are some examples of translation requests using different values of top_p:

• Input: “What is your name?” (English)
• Output with top_p = 0.1: “¿Cómo se llama?” (Spanish)
• Output with top_p = 0.5: “¿Cuál es su nombre?” (Spanish)
• Output with top_p = 0.9: “¿Qué nombre tiene?” (Spanish)

• Input: “Je m’appelle Pierre.” (French)
• Output with top_p = 0.1: “My name is Pierre.” (English)
• Output with top_p = 0.5: “I’m called Pierre.” (English)
• Output with top_p = 0.9: “I go by Pierre.” (English)

Here is an example of a translation request and some possible outputs using different values of top_p:

• Input: 私は猫が好きです。
• Output with top_p=0: I like cats.
• Output with top_p=0.5: I love cats.
• Output with top_p=0.9: I am fond of cats.

We can see that as top_p increases, the output becomes more diverse and less literal. However, this also means that some outputs may not be accurate or natural. Therefore, we need to balance between diversity and quality when choosing the value of top_p.

We can also use nucleus sampling to translate sentences from other languages to English, such as Korean to English. Here is another example:

• Input: 저는 책을 읽는 것을 좋아합니다.
• Output with top_p=0: I like reading books.
• Output with top_p=0.5: I enjoy reading books.
• Output with top_p=0.9: Reading books is my hobby.

Breakdown nucleus sampling works

To illustrate how nucleus sampling works, let’s consider an example of generating text after the prompt “I want to eat”. Suppose we have a language model that assigns the following probabilities to some possible next words:

If we use greedy search, we would always pick the word “a” as the next word, which may not be very diverse or interesting. If we use top-k sampling with k=3, we would only sample from the words “a”, “some”, and “pizza”, which may exclude some relevant words like “salad” or “sushi”. If we use nucleus sampling with p=0.6, we would sample from the smallest set of words whose cumulative probability is at least 0.6, which in this case would be {“a”, “some”, “pizza”, “salad”}. This set has a total probability of 0.65, and we would sample from it according to their relative probabilities. For example, the word “a” would have a 0.3/0.65 = 0.46 chance of being picked, while the word “salad” would have a 0.05/0.65 = 0.08 chance. This way, we can generate more diverse and relevant text than using greedy or top-k sampling.

Nucleus sampling can be applied to any language generation task, such as translation, summarization, or dialogue. For example, if we want to translate the sentence “I want to eat” from English to French, we can use nucleus sampling to generate possible translations. Suppose our translation model assigns the following probabilities to some possible next words in French after the prompt “Je veux”:

If we use nucleus sampling with p=0.7, we would sample from the set {“manger”, “boire”, “aller”}, which has a total probability of 0.7. Then, depending on the next word we sample, we can continue generating the rest of the translation using nucleus sampling again. For example, if we sample the word “manger”, we can generate translations like:

• Je veux manger une pizza.
• Je veux manger de la salade.
• Je veux manger du sushi.
• Je veux manger un gâteau.

If we sample the word “boire”, we can generate translations like:

• Je veux boire de l’eau.
• Je veux boire du café.
• Je veux boire du vin.
• Je veux boire une bière.

If we sample the word “aller”, we can generate translations like:

• Je veux aller au cinéma.
• Je veux aller à la plage.
• Je veux aller au restaurant.
• Je veux aller dormir.

As you can see, nucleus sampling can produce diverse and fluent translations that capture the meaning of the original sentence.