Generally speaking, a protest in the sense relevant here is “a usually organized public demonstration of disapproval” (of some law, policy, idea, or state of affairs), while a riot is “a disturbance of the peace created by an assemblage of usually three or more people acting with a common purpose and in a violent and tumultuous manner to the terror of the public” (both according to Merriam-Webster’s Dictionary of Law).
The term protest as used above is not formally defined in the U.S. Code, as far as I know. The federal Anti-Riot Act (1968) defines the term riot in part as “a public disturbance involving…an act or acts of violence by one or more persons part of an assemblage of three or more persons, which act or acts shall constitute a clear and present danger of, or shall result in, damage or injury to the property of any other person or to the person of any other individual” (Title 18, Chapter 102, §2102). The Act prohibits interstate travel or the use of any “facility” of interstate commerce, including “the mail, telegraph, telephone, radio, or television,” to incite or participate in a riot or to abet another person in doing so (§2101). Many states and municipalities have their own laws against rioting and incitement to riot based on similar definitions, while others prohibit the criminal elements of rioting (e.g., destruction of property, arson, looting, assault, disorderly conduct, disturbing the peace, unlawful assembly) under separate laws against those crimes.
The right to engage in peaceful protest is, of course, protected by the First Amendment. Nevertheless, the Supreme Court has established that governments may limit protests through what are called “time, place, and manner” restrictions, provided that the restrictions are content- or viewpoint-neutral, narrowly tailored to serve a significant government interest (such as public safety), and so formulated as to allow demonstrators ample alternative channels by which to communicate their messages.