Did Claude's quality drop recently?
21 points by MeetingsBrowser 13 hours ago | 14 comments
At least subjectively, Claude has been outperforming other LLMs for me for a while.

They seemed to be having significant availability issues this week.

As of this morning things are working for me but the quality of the responses seems to be much worse.

I frequently ask for lists of ideas and then drill down in follow up prompts. Today Claude keeps responding to my request for bulleted lists with something like, “I should not use lists unless explicitly requested and should instead write in paragraphs”, and then responds in paragraphs.

Is anyone else having a similar experience?

  • philshem 12 hours ago |
    Yes, there are some reports. For example: https://news.ycombinator.com/item?id=42215912
    • MeetingsBrowser 12 hours ago |
      Those are mostly about the availability issues.

      I’m not having trouble getting responses as of today, but the quality of the responses seems to be much worse.

      • wswope 8 hours ago |
        The underlying implication of the linked comment is that Anthropic is using quantization or similar quality-reducing strategies to help keep their service online due to the same shortage that has been causing availability issues.
  • wqaatwt 8 hours ago |
    Yes. It seems to be almost incapable of communicating in anything but terse (up to 5 word or so) bullet points.

    And even when you force it write in coherent sentences the output still seems markedly worse than it used to.

    • blackeyeblitzar 8 hours ago |
      I wonder if it is falling back to concise mode as a means to handle load?
      • wqaatwt 8 hours ago |
        When I switch it back to full I just get longer bullet point lists so perhaps they are doing that silently.
  • patrickhogan1 6 hours ago |
    Well, first off there is no such thing as Claude as there are multiple models that you can select from. You did not list which model you were using. In my opinion the Claude 3.5 Sonnet model is spectacular. It’s the best model yet for coding both on leaderboard and empirically in projects I’ve had it help me with.

    This topic is discussed in recent Lex Fridman interview with with CEO of Anthropic where he very clearly walks through how these claims of it being dumber or not true. It’s a great interview and after listening to it I’m even more bullish on Anthropic.

    There was a small degradation in performance that they posted an alert at the top of the page 2 nights ago. It didn’t affect the quality of the responses I got but it didn’t cause somewhat of a slowdown in response speed.

    • MeetingsBrowser 5 hours ago |
      > Well, first off there is no such thing as Claude as there are multiple models that you can select from.

      Apologies, I assumed people would infer that I am referring to 3.5 sonnet.

      > In my opinion the Claude 3.5 Sonnet model is spectacular.

      Mine as well, until this morning.

      > There was a small degradation in performance … 2 nights ago. It didn’t affect the quality of the responses I got…

      Also same, but as of this morning the performance is fine but the quality seems to have gotten worse.

      > This topic is discussed in recent Lex Fridman interview with with CEO of Anthropic where he very clearly walks through how these claims of it being dumber or not true

      Could you elaborate on what was said?

      • MeetingsBrowser 5 hours ago |
        I found the interview [1].

        TL;DR they don’t change the weights, but they sometimes run A/B tests and modify the system prompt. The underlying model is very sensitive to changes. Even a small change can have broad impacts.

        [1]: https://lexfridman.com/dario-amodei-transcript#chapter8_crit...

        • patrickhogan1 36 minutes ago |
          I hope you get it figured out!

          One thing that has helped me when I can’t quickly get to the expected result is using the Anthropic prompt generator in the dev console.

          This isn’t a critique of your prompt—it’s likely solid since you use the system frequently. However, for troubleshooting, the prompt generator can be useful because it creates very long and specific prompts. You can compare the results from your prompt to the ones generated to see where there might be differences.

  • sk11001 6 hours ago |
    I think so, it was bad enough for me to cancel my subscription.
  • ldjkfkdsjnv 5 hours ago |
    The new model is almost certainly a cheaper version of the older model, where they tried to maintain quality.
    • quibono an hour ago |
      Interesting, but isn't the older model still available?
  • muzani 5 hours ago |
    Sounds like it might be switching to Claude Haiku instead of Claude Sonnet.

    Sonnet 3.5 always has this issue for me though. It excessively follows the original instructions, even in vague ways. It's likely 3.5 (new) is even worse. We use 3.0 in production because of this one quirk.