X日报 · 综合

Train your own LLM from scratch!

A step-by-step repo that walks you through building and training a transformer model from scratch using PyTorch. From downloading training data all the way to generating text.

The architecture is built from the ground up following the original "Attention is All You Need" paper. MLP, single head attention, multi-head attention, transformer blocks, and the full transformer model - all coded and explained with detailed diagrams at each step.

Training data comes from The Pile - a diverse 825GB open-source dataset covering books, articles, code, websites, and more. The repo includes scripts to download it, preprocess and tokenize it using tiktoken, store it in HDF5 format, and feed it into training batches.

You can train a 13M parameter model on a single Colab T4 GPU. At 13M parameters the model starts generating proper grammar and coherent short sentences. For billion-parameter training you need at least an A100 or RTX 4090. The repo includes a full GPU compatibility table so you know exactly what's possible on your hardware.

Includes a complete SFT and RLHF guide as a separate notebook for taking your trained model further.

Key capabilities:

• End-to-end pipeline: data download → preprocessing → training → text generation
• Full transformer implementation from scratch with PyTorch
• Trains models from 13M to 2B+ parameters on a single GPU
• Training data from The Pile (825GB, 22 diverse datasets)
• Tokenization via tiktoken (r50k_base)
• SFT and RLHF guide included

100% open source.

I've shared the link in the replies!

Qwopus 3.6 27b-Coder is now live!

Scores a 67% on a full run of SWE bench verified with thinking completely disabled! Q5_K_M

This model is lightning fast for dense class! With a natively finetuned MTP head, it achieves 100 tps on a single 5090! The biggest upgrade here, though, is its stability in programming and tool calling within @NousResearch Hermes agent, with thinking off!

Wall time is crazy fast this way, which makes Hermes feel "native" and snappy, like they were meant for each other. The freedom of running without thinking at all makes you part of the thinking process, and you never get caught waiting 15 minutes for it to finish a thought string, like with the base models.

Thinking on and temp high, .9-1 seems to produce really incredible design and svg results. I reran the Boat survival prompt through a few turns, thinking on, and it seemed to render more fancy models in HTML canvas, but it was much more of a start-a-prompt and wait experience vs the snappy and active iteration with it disabled. It may be worth turning it off and on throughout the build process if you want to get really creative with design.

Really looking forward to seeing how this one performs for y'all! Please post comments with your opinions and use cases below! As always with our fine-tunes, mess with the temperature setting, and run them much hotter than the base!

Please check out the Boat Survival game I posted yesterday, made in 12 turns using Hermes and this model, with thinking off. Link below!

Full swe bench repo-specific breakdown also posted in the comments for those interested!

Happy building, everyone! We're looking forward to your thoughts! Quants uploading now!

https://t.co/kxJE3C39ZZ

We released North Mini Code, our first open-source coding model, just two days ago. We’re already seeing developers using it to build amazing things ⬇️

🚨 SOMETHING VERY STRANGE IS HAPPENING

SpaceX will go public tomorrow at a $1.75T valuation.

The biggest IPO in market history.

And Wall Street just changed the rules right before it happens.

I've been trading for more than 15 years and have never seen them rewrite the rules so urgently:

IPO access now lowered from $500,000 to $2,000 (-99.6% cut).

That means millions of investors can suddenly enter a deal and buy shares tomorrow.

One day before the most expensive IPO in history.

And suddenly...

SpaceX reserved up to 30% of the deal for regular investors.

Three times the normal share.

Why?

Because retail investors need to buy what insiders sell.

And here is the part most people are missing:

SpaceX does not just create demand for SpaceX.

It pulls liquidity out of everything else:

- Retail sells stocks to chase the IPO.
- Funds sell stocks to prepare for forced buying.
- Brokers open access to generate demand.
- Everyone needs cash at the same time.

That is why the market is selling now.

First, insiders create the hype.

Then brokers open the gates.

Then regular investors rush in.

And by the time the crowd realizes what happened, the exit door is already closed.

We’ve seen this before.

2000:

Dotcom IPOs became the symbol of the bubble.

Then Nasdaq collapsed 80%.

2021:

SPACs, Coinbase, Robinhood, Rivian.

Retail thought they were buying the future.

They were buying the exit.

Now the same playbook is back.

Only this time, it is much bigger.

When Wall Street cuts the entry ticket from $500K to $2K right before a $1.75T IPO, they are not giving retail a gift.

They are creating buyers.

Remember:

Insiders need liquidity.

Funds need allocation.

The market needs a dream.

And Wall Street needs someone to hold the bag.

That is what tomorrow is really about.

Reminder: I’ve called all the market tops and bottoms for the last 15 years, including the Bitcoin bottom at $16,000 and the top at $126,000.

The next call will be even more important.

When I exit the markets completely, I’ll post it here publicly like I always do.

Turn notifications on. If you’re not following yet, you’ll understand why that was a mistake later.

Stock market crash 1995 vs 2026

Same pattern:

1995: Huge overvalued Palm IPO -> Dot-Com Bubble crash

2026: Huge overvalued SpaceX IPO -> AI Bubble crash coming

Remember that I was first to warn you https://t.co/fsrj53T0J5

Before SpaceX IPOs, let's talk about space.

Join us with @temaetfs to talk about space and investing regarding their new ETF, $NASA.

Join tomorrow at 1030AM EST:
https://t.co/kdDP21m0Em https://t.co/rgq8v3l2iO

Peter Lynch's favorite metric is the PEG Ratio. A PEG ratio under 1 means a company is growing faster than its valuation implies.

Here are 10 semiconductor stocks with a PEG under 1 right now:

1. $COHR - Coherent (PEG ~1.0x)

Q3 revenue $1.81B, up 21% YoY. Datacenter segment hit $1.4B with a book-to-bill above 4x. Ramping 1.6T optical transceivers in H2. The vertically integrated AI connectivity play.

Something tells me Trump is going to put about every single positive thing out possible before the largest IPO in history tomorrow.

It could be an unreal day in the market tomorrow.

Remember, any insiders that sell will then have a shit ton of capital to put to work in other parts of the market

VPEC new price hikes on Epiwafers today.

Positive bottleneck read through on companies like $IQE and Landmark (3081) in terms of pricing power/demand for epiwafers.

This follows $MTSI investment into IQE to secure capacity, and shows how important some of these chokepoints are.

(disclosure: have positions in IQE)

Try it out

Good interview https://t.co/IoiIJPLnXG

This is what holding Ethereum for a year does to you: https://t.co/HVnZfNyY1N

Elon Musk becomes the world’s first trillionaire today. https://t.co/J8GuyNPFy7

JUST IN: South Korea sentences ex-President Yoon Suk Yeol to 30 years in prison over drones allegedly sent into Pyongyang to justify marital law.