Initial import from GitHub
This commit is contained in:
@@ -0,0 +1,58 @@
|
||||
# Empirical Results
|
||||
|
||||
## 2025 Controlled Experiments
|
||||
|
||||
A controlled study published in December 2025 measured output truncation across several frontier models, including GPT-4 variants and DeepSeek. Three experiments were conducted:
|
||||
|
||||
### Experiment A: Multi-Part Instruction Compliance
|
||||
|
||||
Models were given complex prompts with multiple explicit requirements (formatting constraints, length requirements, mandatory sections). Results:
|
||||
|
||||
- No model fully satisfied both length requirements and all sub-part instructions natively
|
||||
- Models frequently omitted mandatory output sections
|
||||
- Required formatting constraints were routinely skipped
|
||||
- Explicit length requirements were consistently undershot
|
||||
|
||||
### Experiment B: Decoding Suboptimality
|
||||
|
||||
Tested whether truncated outputs resulted from suboptimal token selection (the model "knowing" the right answer but selecting a worse token). Results:
|
||||
|
||||
- Limited evidence of decoding suboptimality on simple reasoning tasks
|
||||
- The model's greedy, truncated output generally aligned with its highest-confidence solution
|
||||
- Truncation is a deliberate behavioral choice, not a decoding failure
|
||||
|
||||
### Experiment C: Context Degradation
|
||||
|
||||
Tested whether models lose track of instructions during long, multi-turn conversations. Results:
|
||||
|
||||
- Surprising resilience against context degradation during 200-turn conversational tests
|
||||
- Models maintained key facts and instructions significantly better than hypothesized
|
||||
- Context loss is not the primary cause of truncation
|
||||
|
||||
### Key Conclusion
|
||||
|
||||
Laziness is not a failure of memory, context processing, or core model capabilities. It is a behavioral artifact triggered by:
|
||||
1. Instruction complexity exceeding internal effort thresholds
|
||||
2. Aggressively calibrated stopping pressure
|
||||
3. Economic constraints embedded in the alignment layer
|
||||
|
||||
## Prompt Stimulus Effectiveness (Microsoft Research)
|
||||
|
||||
Controlled testing of psychological prompt stimuli documented in a Microsoft Research study:
|
||||
|
||||
| Stimulus | Measured Effect |
|
||||
|:---|:---|
|
||||
| Financial incentive framing ("$200 tip") | +45% output quality and length |
|
||||
| Step-by-step instruction ("take a deep breath") | Accuracy: 34% to 80% on logic tasks |
|
||||
| Stakes framing ("critical to my career") | +10% average performance |
|
||||
| Combined (multiple stimuli) | Up to +115% overall performance |
|
||||
|
||||
These effects are reproducible and stem from statistical correlations in the training data between stakes language and high-effort human outputs.
|
||||
|
||||
## Seasonal Output Variation
|
||||
|
||||
Statistical analysis of ChatGPT outputs during November-December 2023 versus January-March 2024 confirmed:
|
||||
|
||||
- Measurable decrease in average output length during December
|
||||
- Correlation with reduced work output in the training data during holiday periods
|
||||
- Output length increased when the system prompt explicitly stated a non-winter month
|
||||
@@ -0,0 +1,20 @@
|
||||
# References
|
||||
|
||||
## Cited Studies
|
||||
|
||||
- **EmotionPrompt (Microsoft Research)** — Demonstrates that emotional and stakes-based prompt framing mathematically improves LLM reasoning quality and output length. Documents the +45% improvement from financial framing and +115% from combined stimuli.
|
||||
|
||||
- **LazyBench** — Proves that frontier models (Gemini 1.5 Pro, GPT-4o) actively select cognitive shortcuts and fail tasks they are capable of solving when the perceived effort exceeds internal thresholds.
|
||||
|
||||
- **Compounding Error Avoidance** — Research demonstrating that models truncate outputs as a risk mitigation strategy, preferring shorter responses to reduce the surface area for factual errors on long-form tasks.
|
||||
|
||||
- **Seasonal Behavior Analysis (Winter Break Hypothesis)** — Statistical analysis confirming that LLMs internalize seasonal work patterns from training data, producing measurably shorter outputs during periods corresponding to human holiday seasons.
|
||||
|
||||
- **2025 Controlled Laziness Experiments** — Three-part academic study (December 2025) confirming that output truncation is a behavioral artifact of alignment training, not a failure of context processing or model capability.
|
||||
|
||||
## Further Reading
|
||||
|
||||
- Google Gemini API documentation on `thinking_level` parameter configuration
|
||||
- Anthropic MCP (Model Context Protocol) specification and integration guides
|
||||
- OpenAI API reference for temperature and Top-p parameter tuning
|
||||
- YAML front-matter specification for SKILL.md lazy-loading architecture
|
||||
Reference in New Issue
Block a user