New York City MyCity chatbot advises businesses to break the law
- occurred
- 2024-03-29
- reported
- 2024-03-29
- location
- New York, NY, US
- domain
- government
- agent system
- NYC MyCity business chatbot (Microsoft Azure powered)
- reporter
- VAIDDB editorial
- origin
- editorial
== SUMMARY ==
A New York City government chatbot for small businesses told users that landlords may refuse Section 8 tenants and lock tenants out, and that there are no rent restrictions — all contrary to city law.
== DESCRIPTION ==
The Markup tested the MyCity business chatbot launched by the Adams administration in October 2023. It found the bot asserting that landlords are not required to accept tenants on rental assistance (source-of-income discrimination is illegal in NYC), that there are no restrictions on residential rent increases, that a landlord could lock out a tenant, and that restaurants could serve cheese nibbled by a rodent.
The city acknowledged that answers were "wrong in some areas" but kept the chatbot online as a beta, adding stronger disclaimers telling users not to treat responses as legal or professional advice. Because the tool carried the city's authority, businesses acting on its answers risked violating housing and consumer law.
== ROOT CAUSE ==
A generative assistant was deployed for legally binding guidance without grounding in, or verification against, the applicable municipal law.
== MITIGATION ==
The city added beta and accuracy disclaimers, linked to authoritative sources and urged verification, but left the service running.
== REFERENCES ==
- news NYC's AI chatbot was caught telling businesses to break the law. The city isn't taking it down — AP News, 2024-04-02
- news NYC's AI Chatbot Tells Businesses to Break the Law — The Markup, 2024-03-29
== INVESTIGATION ==
no investigation recorded.
== VAID LEVEL 2: INCIDENT ==
Significant failure of a safety provision with no actual harm.
- Failure of one control layer (e.g. tool permission, rate limit, filter).
- Unintended side effects contained to a single system or tenant.
- Exposure of limited internal data without external propagation.