Georeactor Blog

I continue to rework the "Chat-OMG" and "Comments / Criticism" sections of "ML Arxiv Haul" into this new series.

It's been two months since my last "Ignore Previous Instructions" and about four since the ML Arxiv dumps. I have dozens of papers listed somewhere going unread. I don't know how I want to manage it yet.

Topical Issue: Head Honcho

Emad Mostaque, the CEO of Stability AI, resigned last month. Unflattering profiles appeared in Forbes and Fortune. The whole arc of Stability AI suddenly was in question. Who was Emad before this, was he just throwing his money around, did the original work come from Runway, when did he start talking about crypto and decentralization..? In a community of ML researchers and hobbyists, they loved Stable Diffusion and didn't mind if Emad said something weird on a podcast and talked shit on Twitter. But office politics aside, at the end of the day it does matter who VCs are willing to fund.
IMO this whole thing was also chaotic from the beginning (I attempted to work with Carper AI, a code-generation team which they picked up in late 2022, and the whole conversation had to happen on their Discord; they've released only one model after May 2023).

After plagiarism reviews swept northeast academia in January, the Dana-Farber Cancer Institute talked about correcting multiple controversial papers.
Their CEO's paper was retracted this month, 10 years after questions were first raised: https://twitter.com/addictedtoigno1/status/1781093982086332821

After Harvard president Claudine Gay resigned from that position, there was a sensible response by a Harvard virologist https://twitter.com/KizzyPhD/status/1742280103373996076 which the Twitterverse also attacked. Many people there seem to think a virology lab is a class where you should have facts read to you out of a textbook?

I'm not sure what I can say about all of these. It's good to have a former researcher in charge of your institution and awkward for them to have made past mistakes, even if it has nothing to do with what they do now.
Part of the disconnect is that research leaders often are from generational postgraduate education and/or generational wealth. There are entire fields which I didn't understand until someone explained for example Urban Planning is organized as a graduate program. There are these 'summer schools' for ML and quantum computing which seems to be exchanging real information and building temporary communities, without the hype people bring to research conferences, and honestly I still don't know much about how those work culturally or why they seem to be more common in Europe.

Research in Turmoil

A quantum algorithm paper was published showing a possible attack on lattices. It looked like maybe future research would extend this to the lattice-based algorithms used in Post-Quantum Encryption. Luckily a bug was reported in the code.

ChatGPT-y words have been appearing much more often in peer review: https://arxiv.org/abs/2403.07183

Epoch AI rewrites Chinchilla scaling laws https://epochai.org/blog/chinchilla-scaling-a-replication-attempt

I can't follow this narrative as well, but an acclaimed preregistration study appears to have failed, or at least not revealed where the studies' hypotheses were preregistered. https://statmodeling.stat.columbia.edu/2024/03/27/the-feel-good-open-science-story-versus-the-preregistration-who-do-you-think-wins/

LLMs on the Loose

Wizard 2 models from Microsoft were quickly withdrawn for further safety reviews, but now their older models, finetuning dataset, and GitHub Pages repo are missing. Is it possible that the Wizard and safety teams didn't know about each other?
https://www.reddit.com/r/LocalLLaMA/comments/1cbzc71/where_is_wizardlm_are_we_ever_going_to_get_the/

The new Meta AI will join Facebook discussions for unclear reasons, claiming to have a kid in the school https://twitter.com/edzitron/status/1781825480179741056

Adobe's alternative to Midjourney and other copyright-laden image models, is itself partially trained on Midjourney: https://twitter.com/rachelmetz/status/1778787200978891043

The Claude model refuses to display people and dinosaurs together for safety reasons: https://twitter.com/emollick/status/1771358841751454010/photo/3

A Netflix "true crime" documentary upscaled or otherwise edited some of their photos with AI https://futurism.com/the-byte/netflix-what-jennifer-did-true-crime-ai-photos

Some aspects (particularly characters) from Midjourney look a lot like their counterparts in OpenAI's generated videos: https://twitter.com/nickfloats/status/1758496957591695821

Strange marketing move by Washington's state lottery backfires with inappropriate AI image https://mynorthwest.com/3956403/rantz-washingtons-lottery-ai-porn-user/

A library is acquiring AI-generated books - comments suggest the author is using the library lending service to request they acquire print copies https://mastodon.social/@rmounce/112207921381671869

"Polaris is the first safety-focused LLM for real-time patient-AI healthcare conversations" - from Hippocratic AI. They appear to have a Mixture-of-Experts LLM. Seems like the main sensitive area for an unknown startup to jump into (such as decting toxic dosages of OTC medications) https://twitter.com/arankomatsuzaki/status/1770636544434729130

An official NYC chatbot, which was made available last year, was shown to give bad and illegal advice to renters and landlords. Then people tried jailbreaking it with prompts. It will be kept online anyway. https://arstechnica.com/ai/2024/03/nycs-government-chatbot-is-lying-about-city-laws-and-regulations/

Not to go full Dead Internet Theory, but in recent Reddit threads about ML or TV shows, I quickly got a response paraphrasing what I'd said, or adding something vague and contradictory. Why ask a question just to paste the responses into ChatGPT? Just engagement mining?

Clever LLM use

Prompting ChatGPT with "no yapping" seems to avoid overly wordy or contextualized responses.

Finding if Star Trek or other story prompts improve accuracy on problem sets https://twitter.com/emollick/status/1778268671896400029

Using OpenAI API to add custom biases and return more than the top 5 probabilities: https://twitter.com/jxmnop/status/1765873234489061539

Finding small edits within The Great Gatsby: https://twitter.com/emollick/status/1765526220639440964