I like that with #databricks I can define cluster timeout periods so that when I space off and leave it running, it shuts the cluster down so I don't burn through resources too quickly.

*starts cluster*
Looks away. Pets the dog, gets a glass of water, checks the garden, etc, sits back down to see my the cluster terminating due to idle settings.

Ha, I'm such a dumb bitch. Let's try this again.

*starts cluster*
Oh, out of water....

Databricksのテーブルで頻繁に実行されるクエリーの特定
https://qiita.com/taka_yayoi/items/5b6571381778f46a8c09?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items
#qiita #Databricks #UnityCatalog

The #databricks #AI assistant is so bad. Seems like they're using a worthless #LLM and also doing a really bad job of stuffing the context

生成AIでデータ分析やコーディングはどう変わるのか？
https://qiita.com/taka_yayoi/items/72b157b2904d170ef165?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items
#qiita #Databricks #LLM

Spark vs SQL: When should you choose Spark?

Spark: Big jobs that need to do table scans

SQL: Lots of small single-record reads and writes, like a CRUD app

Spark saturates compute and memory resources, so it can be very cheap for processing large amounts of data, whereas SQL optimizes for the noisy neighbor problem and tries to ensure that queries don't impact the performance of other concurrently running queries. Spark only allows one query at a time.

#databricks #data #dataviz #datascience

ジョージア工科大学でコンピュータサイエンスを学び始める話
https://qiita.com/kohei-arai/items/6da2327165dc4deeb146?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items
#qiita #Python #ポエム #大学院 #Databricks #リスキリング

FastAPIを使ってストリーム対応のLLM API RESTサーバを作る on Databricks
https://qiita.com/isanakamishiro2/items/c55123343e95fd2769a6?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items
#qiita #Databricks #FastAPI #langchain #LLM #CTranslate2

New blog post: How to integrate Kedro and Databricks Connect 🔶

In this blog post, our colleague Diego Lira explains how to use Databricks Connect with Kedro for a development experience that works completely inside an IDE.

https://kedro.org/blog/how-to-integrate-kedro-and-databricks-connect

Install it with

```
pip install databricks-connect
```

#kedro #python #pydata #datascience #databricks #dbx #spark #pyspark

databricks is now a supported by PipeRider for dbt projects!

Now databricks users can get:

- Data Impact Reports
- Data profile comparisons
- Pull Request Impact summaries
- Lineage Diff

Check the PipeRider docs for a full list of supported data sources:

https://docs.piperider.io/reference/supported-data-sources

#dbt #piperider #databricks #datyaquality #dataimpact #dataengineering #analyticsengineering #opensource

Databricks Certified Data Engineer Professional 合格体験記
https://qiita.com/shotkotani/items/1eec52bae87a5ff09f48?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items
#qiita #Spark #Databricks #dataengineering #データエンジニアリング #合格体験記

Published yesterday. Covers big UI updates for two #Azure Data Platform services that have been introduced recently.

Contains couple of observations about #Databricks Repos and #Purview.

https://www.kevinrchant.com/2023/08/04/big-ui-updates-for-two-azure-data-platform-services

Remember kids, your #python #databricks job won't execute unless you give it a `if __name__ == "__main__":`

Covers big UI updates for two Azure Data Platform services that have been introduced recently. #Azure #Databricks #Purview

https://www.kevinrchant.com/2023/08/04/big-ui-updates-for-two-azure-data-platform-services

DB-Engines Ranking climbers of the month:
1. #DynamoDB
2. #SnowflakeDB
3. #Databricks
https://db-engines.com/en/ranking

I guess the promising thing about work today is that our department is suddenly having some weird #Databricks initiation script issue after exactly zero environmental or package changes, so all the work I'm supposed to be doing isn't getting done... 😂

Took some time today to rewrite a local build script I have for my #ApacheSpark #Excel data source. I use it for running tests locally against multiple Spark versions and building jar files to deploy to a test #Databricks instance.

The current script is all in #powershell but I spend very little time in there anymore, so rewrote it as a #nushell script, and it’s so much cleaner and nicer to read

日本リージョンでもDatabricksアシスタントが使えるようになりました！
https://qiita.com/taka_yayoi/items/cb958cb0da9fae88affb?utm_campaign=popular_items&utm_medium=feed&utm_source=popular_items
#qiita #Databricks

Hear from Naveen Rao of @MosaicML, a leading provider of LLMs for enterprises that was recently acquired by Databricks for $1.3 billion as he shares his success on the #VBTransform stage at 2:55 pm PST.

#mosaicml #databricks #venturebeat #llm #enterprise #ml #machinelearning #press

New blog post: How to use Databricks managed Delta tables in a Kedro project 🔶

In this post our colleague Jannic Holzer explains how to use a newly-released dataset for managed Delta tables in Databricks within your Kedro project.

https://kedro.org/blog/managed-delta-tables-kedro-dataset

Install it with

```
pip install "kedro-datasets[databricks.ManagedTableDataSet]"
```

#kedro #machinelearning #datascience #databricks #spark #python #pydata

@josiah @orizuru Tbh, it is not a total presumption. I primarily work in #rstats and have faced the same issues.

The DevOps team told the same to one of my teammates. And the Plumber library had some authentication issues related to UMS2 IIRC.

We adapted databricks quickly and found a way to call the R scripts from a Python notebook.

Even on #Databricks, the support of R is so poor!