NYC's MyCity chatbot told ten Markup staffers landlords can refuse Section 8 vouchers

chat.nyc.gov, October 2023

On October 16, 2023, Mayor Eric Adams and Chief Technology Officer Matthew Fraser announced the New York City Artificial Intelligence Action Plan, which the city's release called "first-of-its-kind." At the same event, Adams and Small Business Services Commissioner Kevin D. Kim launched MyCity Business. Its centerpiece was a chatbot on chat.nyc.gov, described by the release as drawing on "trusted information from more than 2,000 NYC Business web pages" and offering "actionable and trusted information" on compliance and violations.¹

The bot's own page carried a "Beta" tag. Under the input box a line of small text warned that the tool "may occasionally produce incorrect, harmful or biased content." Above that line, larger text called the bot "trained to provide you official NYC Business information."²

For five months, the chatbot answered.

Ten Section 8 queries, one answer

On March 29, 2024, The Markup's Colin Lecher published the results of a set of reproduced prompts, co-published with THE CITY and Documented.² Asked whether buildings were required to accept Section 8 vouchers, the bot returned "no, landlords do not need to accept these tenants." One earlier Markup query on the same question had returned the correct answer. When ten separate Markup staffers asked it, the bot gave all ten the wrong answer. That is the closest thing on record to a rate. The same query did not always return the same reply from this bot, so every quoted response is one query's return.²

The Markup ran other prompts. Asked about a cashless restaurant, the bot said "Yes, you can make your restaurant cash-free" and told the reporter "There are no regulations in New York City that require businesses to accept cash as a form of payment." Asked about rent, it said "there are no restrictions on the amount of rent that you can charge a residential tenant." Asked about a business taking a cut of workers' tips, the bot said it was fine. Asked about locking out a tenant, the bot said it was legal. Each of those answers contradicted a specific law or agency guidance that the city itself publishes.²

Rosalind Black, Citywide Housing Director at Legal Services NYC, called the housing errors "dangerously inaccurate" and said the bot "should be taken down." Andrew Rigie of the NYC Hospitality Alliance said a business owner had already alerted him and he had seen the errors himself. Leslie Brown, spokesperson for the NYC Office of Technology and Innovation, said the pilot "has already provided thousands of people with timely, accurate answers" and that the city would keep upgrading it. A Microsoft spokesperson declined to comment.²

Adams keeps the chatbot live

At an April 2, 2024 press conference, Adams was asked whether the city would take the chatbot down. He said: "We're identifying what the problems are, we're gonna fix them, and we're going to have the best chatbot system on the globe." His Chief Advisor Ingrid Lewis-Martin compared the tool to early MapQuest: "Bad things were happening. But now MapQuest is almost perfected. Same thing."⁶

At a March 2024 New York Law School panel The Markup obtained a recording of, SBS Commissioner Kim had called the bot a "baby step" and said "trust is the most important thing" for a government using AI. He also told the room: "This AI chatbot not only has garnered trust, it also can service people 24/7 without being put on hold." Rakesh Malhotra, co-founder and managing partner of Nuvalence, the tool's integration partner, described the SBS approach as "How do we get to yes?" He added that the technology "comes with some risk in being at the forefront of this."⁶

The city changed the disclaimer. "May occasionally produce incorrect, harmful or biased content" was replaced by a longer note: "Always double-check its information using the provided links or by visiting MyCity Business. Do not use its responses as legal or professional advice or provide sensitive information to the Chatbot." Then The Markup tested the new disclaimer against the bot itself.⁶

A day after that press conference, on April 3, the Associated Press's Jake Offenhartz published a second reporter's tests. The chatbot had told the AP the day before that it was legal to fire a worker for reporting sexual harassment, for not disclosing a pregnancy, or for refusing to cut dreadlocks. It had said businesses could use black garbage bags and did not have to compost, contradicting two city waste initiatives. Asked whether a restaurant could serve a customer cheese a rat had nibbled on, the chatbot answered "Yes, you can still serve the cheese to customers if it has rat bites." It then advised the restaurant to assess "the extent of the damage caused by the rat" and to "inform customers about the situation."⁷

AP obtained a Microsoft statement The Markup had not. The company said it was working with city employees "to improve the service and ensure the outputs are accurate and grounded on the city's official documentation." Julia Stoyanovich, director of the Center for Responsible AI at NYU, told AP: "They're rolling out software that is unproven without oversight. It's clear they have no intention of doing what's responsible." She called the Adams administration's approach "reckless and irresponsible." Suresh Venkatasubramanian of Brown's Center for Technological Responsibility, Reimagination, and Redesign told AP: "If the chatbots are used to replace a person, then you lose accountability while not getting anything in return." Ted Ross, chief information officer in Los Angeles, said that city's chatbots did not rely on large language models and its content was closely curated.⁷

Retrieval, then generation

The chatbot's back end was Azure OpenAI, integrated by Nuvalence, with the 2,000-plus NYC Business web pages as its knowledge store. That is the pattern the paper has covered under retrieval. The system searches an outside store at answer time and puts the closest passages into the prompt. The model then has the right text in view when it writes its reply.⁸

Retrieval does not force the sentence the model then writes to say what those passages say. At generation time, the model is doing what a language model does anywhere else: sampling likely next tokens. The distribution it samples from biases it toward one kind of sentence. When that bias runs against a retrieved fact, the bias can win. This is hallucination in a system where a correct source was in the prompt: the fact was fetched, the generator wrote around it.

The other visible feature of this deployment was that its answers varied between runs. A tenant attorney posted on X that the bot had answered "no" when asked whether tenants could withhold rent for unmade repairs. The Markup then tested the same question twice and got two different replies. One of them, for the first time, volunteered that the tool was "an AI language model" and "not a lawyer."⁶ Sampling from a probability distribution over the next token, at any temperature above zero, walks to different answers on different tries.

Layering a disclaimer over that behavior did not repair it. The bot generated an answer that contradicted the disclaimer as easily as it generated an answer that contradicted the source.

Mamdani takes it down

At a January 30, 2026 press conference on the city's $12 billion budget gap, Mayor Zohran Mamdani said: "The previous administration had an AI chatbot that was functionally unusable. It was costing the administration around half a million dollars." Mamdani's spokesperson Dora Pekec confirmed by text that the administration planned to take the chatbot down; a transition-team member had brought The Markup's earlier reporting to Mamdani. The tool's build reportedly cost nearly $600,000.⁹

On February 4, 2026, the city took the chatbot offline. chat.nyc.gov today shows a single line: "The Chatbot beta test has ended. Please go to NYC.gov to find NYC government information."¹⁰ The tool ran in beta for 27 months.