Section II Ingestion · Caching · Enrichment Figures 2–3

FutureMarket

An autonomous equity-research loop — ingestion, scoring, execution, and self-grading — described from the source

The Wire

Everything the system knows, it knows for free

FutureMarket buys no data. Every price, filing, headline and indicator arrives through a free tier or an open API, and the architecture is shaped by that constraint more than by any other single decision.

Primary module: services/openbb_service.py 380 lines Provider: yfinance via OpenBB SDK

The consequence of a free-data architecture is that rate limiting, not latency, is the governing concern. It shows up everywhere: in the staggered scheduler offsets that keep thirteen jobs from hitting the same upstream simultaneously; in the half-second sleep between batches when the universe service validates several hundred tickers; in the one-second minimum interval enforced by an asyncio lock in front of the news client; and in seven separate in-process caches, no two of which share an implementation.

The data layer is a thin abstraction over the OpenBB SDK, which itself wraps yfinance. The SDK is deliberately not imported at module load. It is lazy-loaded on first property access, with the auto-build environment flag set at import time, and this is what keeps the FastAPI process from taking several seconds to start.

Five fetch paths exist, and each is written twice: a synchronous cached implementation, and a thin async wrapper that pushes it onto a worker thread. This is the dominant pattern across the whole backend — blocking network and numerical work runs in threads, the event loop stays free, and there is no true async client anywhere in the data path.

Symbol normalisation happens at exactly one point, and only when the provider is yfinance: a dot in a class-share ticker becomes a hyphen. It is a two-line function, and it is the only concession in the codebase to the fact that ticker conventions differ between vendors.

Where a provider does not supply a field, the service does one of two things, and the difference matters. For quote spreads it fabricates: when the provider returns no bid or ask — which is the normal case for the free feed — it synthesises a symmetric ten-basis-point spread around the last price. For valuation ratios it declines: market capitalisation, price/earnings, dividend yield, and beta are all returned as explicit nulls.

The second behaviour is more honest and causes more trouble. Downstream, the tool that formats fundamentals for the language model divides the market-cap field by a billion to render it in billions. A null does not divide. The resulting type error is swallowed by a broad exception handler that discards the income and balance-sheet sections it had already built correctly, and returns an error string in their place.1

Rate limiting, not latency, is the governing constraint. Seven caches exist, and no two of them share an implementation.

On the cost of building against free tiers

Figure 2 · Request lifecycle

One fetch, end to end

The path below is the same for all five fetch methods; only the cache key format and the time-to-live differ. Note that expiry is lazy — entries are only checked when read, never evicted in the background, and the cache has no size bound.

Figure 2 Data fetch lifecycle and cache tiers, openbb_service.py
Data fetch request lifecycle through the TTL cache A flow diagram. An HTTP route calls an async wrapper, which offloads to a thread running a synchronous cached fetch. The fetch builds a cache key, checks a time-to-live dictionary, and either returns the cached tuple or lazily loads the OpenBB SDK, calls the provider, normalises the payload, stores it, and returns. A side panel lists the five cache key formats with their time-to-live values, and a lower band shows the indicator computation branch. FastAPI route api/stocks.py async wrapper asyncio.to_thread(...) build cache key f"prices:{sym}:{s}:{e}" key present & not expired? HIT return cached no network call MISS lazy-load SDK first access only provider call yfinance, blocking normalise date → timestamp OHLC → 2dp, vol → int dot ticker → hyphen store (data, expiry) no eviction, no bound CACHE KEYS AND TIME-TO-LIVE quote:{sym} 30 s prices:{sym}:{start}:{end} 30 s technicals:{sym} 60 s news:{sym}:{limit} 300 s fundamentals:{sym} 3600 s Price keys embed the date range, so key count grows without bound in a long-lived process. Process-local. Not shared with Celery workers. INDICATOR BRANCH — technicals only pull 120 days warm-up for SMA-50 bars < 20? raise ValueError RSI(14) 4 dp MACD(12,26,9) histogram used Bollinger(20, 2σ) 2 dp SMA(20) · SMA(50) SMA-50 only if ≥ 50 bars Indicators are computed by the SDK, not by hand; results are read back through OpenBB's generated column names. Quote path only: if the provider returns no bid/ask, a symmetric 10 bp spread is synthesised around the last price.
Figure 2. The fetch lifecycle. The dashed return path from the store step is the write-back into the same dictionary the decision diamond reads. Every branch shown here is process-local: a Celery worker running the same code has an entirely separate cache, so the two schedulers can and do issue duplicate upstream requests for identical data. backend/services/openbb_service.py:16–34 (cache), :42–61 (lazy SDK, symbol normalisation), :63–95 (prices), :132–173 (quote), :282–362 (technicals) · TTLs from backend/config.py:25–28

Self-report

What the service says about itself

A single status route reports the feature matrix, and it is worth reading closely because several of its flags report intent rather than availability. qlib_quant reflects a configuration boolean, not whether the optional package imported successfully. multi_agent likewise. The health route above it performs no checks at all and returns healthy unconditionally.

shell — curl 
$ curl -s http://localhost:8000/api/status | python3 -m json.tool
{
    "status": "operational",
    "version": "1.0.0",
    "uptime_seconds": <seconds since app_start_time>,
    "started_at": <ISO 8601, naive UTC>,
    "current_time": <ISO 8601, naive UTC>,
    "tracked_stocks": [
        "AAPL",
        "TSLA",
        "NVDA",
        "MSFT",
        "GOOGL",
        "AMZN"
    ],
    "database_url": "[redacted — filename withheld from this report]",
    "data_provider": "openbb",
    "features": {
        "stock_data": true,
        "predictions": true,
        "multi_agent": true,        // config flag, not a probe
        "backtesting": true,        // hardcoded literal
        "autonomy_loop": true,
        "openbb_data": true,
        "fundamentals": true,
        "news": true,
        "technicals": true,
        "qlib_quant": true         // config flag; package may be absent
    }
}

$ curl -s http://localhost:8000/api/health
{"status":"healthy", ...}   # no dependency checks are performed

Keys, ordering and literal values are exactly those constructed at backend/main.py:408–436; the tracked-symbol list is the default at backend/config.py:51. Timestamps and the database filename are withheld rather than invented.

The universe

Six hundred and eighty-one tickers, hand-typed

Before anything can be screened, something must define the candidate set. universe_service.py does this from a hardcoded dictionary of eleven sector buckets containing 690 entries, 681 of them unique. Validation applies a minimum market capitalisation of 500 million dollars and a minimum average daily volume of 100,000 shares, with an exchange whitelist.

Three things about this list deserve stating plainly rather than glossing. First, a sector filter is written and then explicitly disabled — the branch that would reject off-target sectors ends in a bare pass, so the twenty-entry target-sector set filters nothing. Second, the list is hand-maintained and shows it: several entries are company names rather than tickers, at least one ticker appears twice under different sectors, and a number of constituents have been delisted or renamed since the list was written. Third, the comment claiming a batched download is inaccurate; inside each batch of fifty, symbols are still fetched one at a time in a serial loop, with a half-second pause between batches.

When live validation returns fewer than a hundred symbols, the service unions in a “quick universe” — the first fifty entries from each sector bucket, deduplicated, about 515 symbols — which discards the market-capitalisation ordering that the fetch had just established. Any exception falls back to that quick universe wholesale and caches it for twenty-four hours.

Filters, as implemented

Market capitalisation
Minimum 500,000,000 — enforced.
Average volume
Minimum 100,000 shares/day — enforced.
Exchange
Whitelist of eight codes — enforced only when the provider supplies an exchange field.
Sector
Twenty-entry target set — declared and then bypassed.

The screener that consumes this service carries its own hardcoded fallback list of roughly 120 symbols, in case the universe service raises. The universe service, in turn, creates its cache directory unguarded in its constructor — which is why every call site wraps construction in a try/except.

Figure 3 · Sentiment

Three sources, renormalised over whichever survive

Sentiment is the most heavily layered part of the ingestion stack, and the only place where a genuine transformer model is used. Weights are fixed. Crucially, the composite renormalises over the sources that returned successfully — so a single available source at nominal 50 per cent weight contributes the entire composite, and a caller cannot tell a one-source reading from a three-source one.

Figure 3 Sentiment fan-in and renormalisation, sentiment_aggregator.py
Three-source sentiment aggregation with weight renormalisation Three source lanes — FinBERT news at weight 0.50, earnings-call tone at 0.30, and SEC filing risk delta at 0.20 — each produce a score between minus one and plus one together with an availability flag. Available lanes are summed with their weights and divided by the sum of available weights, then mapped to a zero to one hundred scale and bucketed into bullish, neutral, or bearish. The news lane is marked as unavailable on the self-fetch path because of a missing method. LANE 1 · WEIGHT 0.50 finbert_service ProsusAI/finbert, CPU-pinned score = p(pos) − p(neg) unavailable on self-fetch path LANE 2 · WEIGHT 0.30 earnings_analyzer management tone ∈ [−1, +1] guidance ±0.15 modifier LANE 3 · WEIGHT 0.20 sec_filing_analyzer risk delta, sign-inverted severity ×0.5 … ×1.25 AVAILABILITY each lane returns (score, available) any exception → available = False score = 0.0 unavailable lanes are dropped, not zeroed Cache: 1800 s, class-level, shared across all instances RENORMALISED WEIGHTED SUM normalised = Σ(sᵢ · wᵢ) / Σ(wᵢ) for i in available lanes only composite = clamp((normalised + 1) × 50, 0, 100) SIGNAL BUCKETS BULLISH composite ≥ 60 NEUTRAL 41 – 59 BEARISH composite ≤ 40 A SECOND, SEPARATE SENTIMENT ENGINE enhanced_sentiment_service.py is unrelated to the above and is what the factor engine actually consumes. keyword lexicon ~70 weighted terms substring match — imprecise analyst rating deltas grade transitions weight 0.20 third-party feed trailing 7 days, cap 30 weight 0.15 news volume — declared weight 0.25, never summed applied instead as a multiplier on the deviation from 50, range ×0.85 to ×1.15 — it amplifies rather than contributes Effective denominator is therefore 0.60, or 0.75 when the third-party feed is configured — not the 1.00 the constants imply. Cache TTL 7200 s. Headlines deduplicated on the first 50 characters of the lowercased title. Two independent sentiment implementations coexist; nothing reconciles their outputs.
Figure 3. Sentiment aggregation. The dashed vermilion outline on lane one marks a live defect: the aggregator calls a method name on the data service that does not exist, so unless headlines are passed in explicitly by the caller, the highest-weighted source is permanently unavailable and the composite reduces to a 60/40 blend of earnings tone and filing risk. The documented seven-day and ninety-day recency half-lives are not implemented — the news lane takes an unweighted mean. backend/services/sentiment_aggregator.py:44–46 (weights), :89–119 (composition), :282–306 (broken fetch) · services/finbert_service.py:36, :117–120 · services/enhanced_sentiment_service.py:121–124, :448–482

Other wires

News, filings, and risk

Open news archive

A dedicated client fetches article lists from an open global news index that requires no key. It maintains a fifty-four ticker map from symbol to company name so that a full-text query can match the company as well as the ticker; unmapped symbols fall back to a bare ticker query, which for a full-text index is very low recall.

Historical queries are approximate by construction and the module says so. The upstream API exposes no start or end parameter, only a timespan measured backwards from now. The service computes that timespan, over-fetches up to the API's 250-record ceiling, then filters by string comparison on the date prefix. Because results are sorted newest-first, older windows are frequently unreachable — the fetch exhausts its budget before it reaches them. This matters more than it appears, because this is the news source used for backtesting.

Requests are serialised through an asyncio lock with a one-second minimum interval, and every failure mode — timeout, HTTP error, non-JSON body — returns an empty list rather than raising.

Risk metrics

The risk service bypasses the provider abstraction entirely and calls yfinance directly. From one year of daily bars it computes beta against a benchmark, thirty- and ninety-day annualised volatility, maximum drawdown over the fifty-two week window, a Sharpe ratio, short interest, recent gap-downs beyond five per cent, and the days remaining until the next earnings date.

These are combined into a hundred-point score where higher means safer — the inverse of the convention used everywhere else in the codebase. The allocation is 25 points to beta, 25 to volatility, 20 to drawdown, 15 to earnings proximity, and 15 to short interest.

Two details are worth flagging. The risk-free rate is a hardcoded constant, annotated in a comment as an approximation of a particular period's Treasury bill rate; it does not update. And in every component, unknown data scores at the midpoint while known-bad data scores at the floor — so a stock with a missing beta scores better than one with a measured beta of 1.5. Missing data is systematically rewarded.

Notes

  1. The practical effect is that the fundamentals block supplied to the language model is usually an error string rather than financial data, and nothing downstream detects this — the extraction stage parses whatever prose comes back regardless. backend/services/agent_tools.py:99–109. ↩