The lawsuit alleges that McDougal was "duped" by her then-lawyer, Keith Davidson, into signing the agreement.
Davidson told her AMI would purchase her story but "not publish it" because of chief executive David J. Pecker's close relationship with Trump.
McDougal gave her story to the National Enquirer in exchange for the money, but the publication engaged in what is known as a "catch and kill", in which the tabloid never actually runs the story. She also asserts that after she spoke with The New Yorker last month after it obtained notes she kept on Mr. Trump, A.M.I. warned that "any further disclosures would breach Karen's contract" and "cause considerable monetary damages".
American Media told The Times last month that it decided not to print McDougal's story because it could not verify important details, but it acknowledged that it discussed her allegations with Cohen, the president's lawyer, saying it did so as part of its reporting process.
"Once she's part of the company, then on the outside she can't be bashing Trump", Pecker told the New Yorker a year ago.
McDougal claims the company paid her $150,000 to remain silent about the affair, which supposedly took place in 2016 after Trump secured the Republican nomination.More news: Radford Highlanders vs. LIU Brooklyn Blackbirds Odds, March Madness Betting Pick
More news: Geelani steps down from TeH post
More news: Police chief says Uber 'likely not' at fault in pedestrian accident
In an email to The New York Times, her new lawyer, Peter K. Stris, accused American Media of "a multifaceted effort to silence Karen McDougal". "Ms. McDougal received $150,000 (nearly half of which went to the lawyer, who she did not realize was colluding with the other side) and a false promise to jumpstart her career as a health and fitness model", McDougal says in her lawsuit.
The decision by Justice Jennifer Schecter of the NY state court in Manhattan in favour of California restaurateur Summer Zervos, a former contestant on NBC's The Apprentice, raises the prospect that Trump might have to answer embarrassing questions in court about his behaviour towards women.
The complaint says that McDougal's own lawyer at the time helped mislead her during the negotiations.
The lawsuit comes as it emerged the porn star Stormy Daniels, who also says she had an affair with Mr Trump, passed a lie detector test over the matter in 2011.
The New York Times reported on Tuesday that McDougal is suing a media company run by a friend of Trump's in order to get out of the 2016 nondisclosure agreement. We are confident that the so-called contract will be invalidated, and are eager for Ms. McDougal to be able to move forward with her life with the privacy she deserves.
American Media publishes The National Enquirer.