[==========] Running 57 tests from 4 test suites.
[----------] Global test environment set-up.
[----------] 31 tests from SecurityITest
[ RUN ] SecurityITest.TestAuthorizationOnListTablets
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:50:33 dist-test-slave-hxxt krb5kdc[423](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:50:33 dist-test-slave-hxxt krb5kdc[423](info): set up 2 sockets
Oct 27 14:50:33 dist-test-slave-hxxt krb5kdc[423](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20251027 14:50:35.649049 416 mini_kdc.cc:121] Time spent starting KDC: real 1.834s user 0.008s sys 0.006s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:50:35 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:50:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:50:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:35.836505 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44309
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44309
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:36.157891 439 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:36.158195 439 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:36.158324 439 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:36.166522 439 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:50:36.166643 439 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:36.166726 439 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:36.166787 439 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:50:36.166870 439 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:50:36.177620 439 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44309
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:36.179159 439 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:36.180523 439 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:36.188526 445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:36.188521 444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:36.189185 439 server_base.cc:1047] running on GCE node
W20251027 14:50:36.189316 447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:36.190620 439 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:36.192068 439 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:36.193275 439 hybrid_clock.cc:648] HybridClock initialized: now 1761576636193170 us; error 90 us; skew 500 ppm
I20251027 14:50:36.193620 439 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:36 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:36.196719 439 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:50:36.198277 439 webserver.cc:492] Webserver started at http://127.0.104.62:36751/ using document root <none> and password file <none>
I20251027 14:50:36.199015 439 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:36.199097 439 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:36.199437 439 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:36.202087 439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd"
format_stamp: "Formatted at 2025-10-27 14:50:36 on dist-test-slave-hxxt"
server_key: "1a22b48f8902441a21dd6a891d07eb1a"
server_key_iv: "d168434157b296d192a6011af933ee31"
server_key_version: "encryptionkey@0"
I20251027 14:50:36.202838 439 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd"
format_stamp: "Formatted at 2025-10-27 14:50:36 on dist-test-slave-hxxt"
server_key: "1a22b48f8902441a21dd6a891d07eb1a"
server_key_iv: "d168434157b296d192a6011af933ee31"
server_key_version: "encryptionkey@0"
I20251027 14:50:36.208434 439 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20251027 14:50:36.211740 454 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:36.213088 439 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:50:36.213230 439 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd"
format_stamp: "Formatted at 2025-10-27 14:50:36 on dist-test-slave-hxxt"
server_key: "1a22b48f8902441a21dd6a891d07eb1a"
server_key_iv: "d168434157b296d192a6011af933ee31"
server_key_version: "encryptionkey@0"
I20251027 14:50:36.213414 439 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:36.236987 439 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:36.240739 439 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:36.241001 439 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:36.260262 439 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44309
I20251027 14:50:36.260280 505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44309 every 8 connection(s)
I20251027 14:50:36.261814 439 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:50:36.266062 506 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:36.270871 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 439
I20251027 14:50:36.271138 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:50:36.273013 416 external_mini_cluster.cc:1468] Setting key 30089ea5a3286e300bf740a3372dc130
I20251027 14:50:36.276234 506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd: Bootstrap starting.
I20251027 14:50:36.279451 506 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:36.280524 506 log.cc:826] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:36.283039 506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd: No bootstrap required, opened a new log
I20251027 14:50:36.289335 506 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44309 } }
I20251027 14:50:36.289651 506 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:36.289750 506 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3446b5e4ddb745808dbb54f6dc8d8ddd, State: Initialized, Role: FOLLOWER
I20251027 14:50:36.290386 506 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44309 } }
I20251027 14:50:36.291100 506 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:50:36.291229 506 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:50:36.291395 506 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:36.294118 506 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44309 } }
I20251027 14:50:36.294747 506 leader_election.cc:304] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3446b5e4ddb745808dbb54f6dc8d8ddd; no voters:
I20251027 14:50:36.295348 506 leader_election.cc:290] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:50:36.295503 511 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:36.296253 511 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [term 1 LEADER]: Becoming Leader. State: Replica: 3446b5e4ddb745808dbb54f6dc8d8ddd, State: Running, Role: LEADER
I20251027 14:50:36.296829 511 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44309 } }
I20251027 14:50:36.297605 506 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [sys.catalog]: configured and running, proceeding with master startup.
Oct 27 14:50:36 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576635, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:36.301614 512 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44309 } } }
I20251027 14:50:36.301905 512 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:36.302244 513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3446b5e4ddb745808dbb54f6dc8d8ddd. Latest consensus state: current_term: 1 leader_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3446b5e4ddb745808dbb54f6dc8d8ddd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44309 } } }
I20251027 14:50:36.302505 513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:36.304438 517 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:50:36.310745 517 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:50:36.313359 508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:36.276995 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37544 (local address 127.0.104.62:44309)
1027 14:50:36.277912 (+ 917us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:36.277929 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:36.279107 (+ 1178us) server_negotiation.cc:408] Connection header received
1027 14:50:36.286444 (+ 7337us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:36.286471 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:36.287002 (+ 531us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:36.287471 (+ 469us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:36.290180 (+ 2709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:36.292620 (+ 2440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:36.293885 (+ 1265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:36.294457 (+ 572us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:36.299784 (+ 5327us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:36.299922 (+ 138us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:36.299944 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:36.300001 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:36.305742 (+ 5741us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:36.306651 (+ 909us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:36.306659 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:36.306665 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:36.306799 (+ 134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:36.307248 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:36.307252 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:36.307255 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:36.308139 (+ 884us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:36.308459 (+ 320us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:36.311484 (+ 3025us) server_negotiation.cc:300] Negotiation successful
1027 14:50:36.311984 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":673,"thread_start_us":434,"threads_started":1}
I20251027 14:50:36.322391 517 catalog_manager.cc:1357] Generated new cluster ID: 90907ac5468c4ce49e9eadea8efbe390
I20251027 14:50:36.322549 517 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:50:36.356750 517 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:50:36.357965 517 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:50:36.367290 517 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3446b5e4ddb745808dbb54f6dc8d8ddd: Generated new TSK 0
I20251027 14:50:36.368101 517 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:36.439396 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44309
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:36.775862 534 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:36.776151 534 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:36.776293 534 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:36.784250 534 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:36.784348 534 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:36.784555 534 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:50:36.795197 534 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44309
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:36.796793 534 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:36.798055 534 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:36.806898 539 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:36.807063 540 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:36.808279 542 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:36.808532 534 server_base.cc:1047] running on GCE node
I20251027 14:50:36.809093 534 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:36.809957 534 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:36.811183 534 hybrid_clock.cc:648] HybridClock initialized: now 1761576636811125 us; error 37 us; skew 500 ppm
I20251027 14:50:36.811476 534 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:36 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:36.814553 534 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:50:36.816066 534 webserver.cc:492] Webserver started at http://127.0.104.1:36297/ using document root <none> and password file <none>
I20251027 14:50:36.816825 534 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:36.816907 534 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:36.817243 534 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:36.820063 534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "60fe7adecee7411e8bce50aba9e03ccd"
format_stamp: "Formatted at 2025-10-27 14:50:36 on dist-test-slave-hxxt"
server_key: "094b18ec366935a76436f53d221d7141"
server_key_iv: "01b6bed46f2c710a58c00297886e6e5b"
server_key_version: "encryptionkey@0"
I20251027 14:50:36.820860 534 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "60fe7adecee7411e8bce50aba9e03ccd"
format_stamp: "Formatted at 2025-10-27 14:50:36 on dist-test-slave-hxxt"
server_key: "094b18ec366935a76436f53d221d7141"
server_key_iv: "01b6bed46f2c710a58c00297886e6e5b"
server_key_version: "encryptionkey@0"
I20251027 14:50:36.826073 534 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:50:36.829309 549 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:36.830610 534 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:50:36.831225 534 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "60fe7adecee7411e8bce50aba9e03ccd"
format_stamp: "Formatted at 2025-10-27 14:50:36 on dist-test-slave-hxxt"
server_key: "094b18ec366935a76436f53d221d7141"
server_key_iv: "01b6bed46f2c710a58c00297886e6e5b"
server_key_version: "encryptionkey@0"
I20251027 14:50:36.831434 534 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:36.863303 534 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:36.867543 534 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:36.867790 534 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:36.868729 534 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:50:36.870218 534 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:36.870308 534 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:36.870409 534 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:36.870471 534 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:36.912729 534 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37029
I20251027 14:50:36.912827 661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37029 every 8 connection(s)
I20251027 14:50:36.914337 534 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:50:36.915103 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 534
I20251027 14:50:36.915324 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:50:36.916036 416 external_mini_cluster.cc:1468] Setting key 236132c61c431f8d4e1cdf1708375b6b
Oct 27 14:50:36 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576636, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:36.936273 664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:36.917410 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:56617 (local address 127.0.104.62:44309)
1027 14:50:36.918029 (+ 619us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:36.918035 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:36.919689 (+ 1654us) server_negotiation.cc:408] Connection header received
1027 14:50:36.921783 (+ 2094us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:36.921811 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:36.921895 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:36.922023 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:36.924415 (+ 2392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:36.925799 (+ 1384us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:36.927008 (+ 1209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:36.927428 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:36.930871 (+ 3443us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:36.930910 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:36.930916 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:36.930956 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:36.933550 (+ 2594us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:36.934213 (+ 663us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:36.934217 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:36.934219 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:36.934303 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:36.934605 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:36.934610 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:36.934614 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:36.934864 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:36.935002 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:36.935689 (+ 687us) server_negotiation.cc:300] Negotiation successful
1027 14:50:36.935939 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":513,"thread_start_us":415,"threads_started":1}
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:50:36.937541 665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:36.918130 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44309 (local address 127.0.104.1:56617)
1027 14:50:36.919004 (+ 874us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:36.919055 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:36.921411 (+ 2356us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:36.922301 (+ 890us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:36.922316 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:36.923189 (+ 873us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:36.924171 (+ 982us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:36.924198 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:36.925968 (+ 1770us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:36.925972 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:36.926871 (+ 899us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:36.926886 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:36.927161 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:36.927933 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:36.927969 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:36.930672 (+ 2703us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:36.933690 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:36.933700 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:36.933719 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:36.934103 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:36.934398 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:36.934402 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:36.934404 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:36.934509 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:36.934978 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:36.934986 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:36.935298 (+ 312us) client_negotiation.cc:769] Sending connection context
1027 14:50:36.935600 (+ 302us) client_negotiation.cc:240] Negotiation successful
1027 14:50:36.936099 (+ 499us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"thread_start_us":380,"threads_started":1}
I20251027 14:50:36.939258 662 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44309
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:50:36.939677 662 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:36.940757 662 heartbeater.cc:507] Master 127.0.104.62:44309 requested a full tablet report, sending...
I20251027 14:50:36.943394 471 ts_manager.cc:194] Registered new tserver with Master: 60fe7adecee7411e8bce50aba9e03ccd (127.0.104.1:37029)
I20251027 14:50:36.945226 471 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:56617
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:36.987591 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44309
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:37.315318 670 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:37.315641 670 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:37.315786 670 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:37.323781 670 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:37.323892 670 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:37.324076 670 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:50:37.334713 670 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44309
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:37.336323 670 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:37.337659 670 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:37.346818 675 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:37.346840 676 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:37.347424 678 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:37.347977 670 server_base.cc:1047] running on GCE node
I20251027 14:50:37.348554 670 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:37.349381 670 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:37.350591 670 hybrid_clock.cc:648] HybridClock initialized: now 1761576637350533 us; error 38 us; skew 500 ppm
I20251027 14:50:37.350860 670 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:37 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:37.353838 670 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:50:37.355275 670 webserver.cc:492] Webserver started at http://127.0.104.2:33805/ using document root <none> and password file <none>
I20251027 14:50:37.355989 670 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:37.356071 670 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:37.356446 670 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:37.358912 670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "fe935882e7054faebbc9669dcc38f736"
format_stamp: "Formatted at 2025-10-27 14:50:37 on dist-test-slave-hxxt"
server_key: "62ac6f8e495dc2520229385e53499448"
server_key_iv: "244712ad4b748c2c47395647f23e2262"
server_key_version: "encryptionkey@0"
I20251027 14:50:37.359669 670 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "fe935882e7054faebbc9669dcc38f736"
format_stamp: "Formatted at 2025-10-27 14:50:37 on dist-test-slave-hxxt"
server_key: "62ac6f8e495dc2520229385e53499448"
server_key_iv: "244712ad4b748c2c47395647f23e2262"
server_key_version: "encryptionkey@0"
I20251027 14:50:37.364746 670 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20251027 14:50:37.368643 685 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:37.369987 670 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:50:37.370141 670 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "fe935882e7054faebbc9669dcc38f736"
format_stamp: "Formatted at 2025-10-27 14:50:37 on dist-test-slave-hxxt"
server_key: "62ac6f8e495dc2520229385e53499448"
server_key_iv: "244712ad4b748c2c47395647f23e2262"
server_key_version: "encryptionkey@0"
I20251027 14:50:37.370332 670 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:37.392115 670 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:37.396152 670 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:37.396456 670 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:37.397877 670 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:50:37.399526 670 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:37.399646 670 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:37.399745 670 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:37.399814 670 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:37.434664 670 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42815
I20251027 14:50:37.434720 797 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42815 every 8 connection(s)
I20251027 14:50:37.436101 670 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:50:37.443053 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 670
I20251027 14:50:37.443259 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:50:37.443730 416 external_mini_cluster.cc:1468] Setting key 488645a46377e878280312747963be62
Oct 27 14:50:37 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:37.457252 800 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:37.439010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:51575 (local address 127.0.104.62:44309)
1027 14:50:37.439469 (+ 459us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:37.439474 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:37.440508 (+ 1034us) server_negotiation.cc:408] Connection header received
1027 14:50:37.442014 (+ 1506us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:37.442020 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:37.442105 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:37.442209 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:37.444426 (+ 2217us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.445969 (+ 1543us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:37.447392 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.447710 (+ 318us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:37.450998 (+ 3288us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:37.451044 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:37.451050 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:37.451088 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:37.454265 (+ 3177us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:37.454957 (+ 692us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:37.454963 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:37.454966 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:37.455062 (+ 96us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:37.455387 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:37.455392 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:37.455395 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:37.455698 (+ 303us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:37.455828 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:37.456644 (+ 816us) server_negotiation.cc:300] Negotiation successful
1027 14:50:37.456892 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":309,"thread_start_us":212,"threads_started":1}
I20251027 14:50:37.458109 801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:37.439585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44309 (local address 127.0.104.2:51575)
1027 14:50:37.440280 (+ 695us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:37.440319 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:37.441735 (+ 1416us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:37.442457 (+ 722us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:37.442466 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:37.443224 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:37.444178 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:37.444204 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.446126 (+ 1922us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:37.446132 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:37.447263 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:37.447279 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.447564 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:37.448352 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:37.448406 (+ 54us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:37.450796 (+ 2390us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:37.454416 (+ 3620us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:37.454424 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:37.454440 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:37.454848 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:37.455178 (+ 330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:37.455181 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:37.455183 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:37.455290 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:37.455817 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:37.455823 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:37.456132 (+ 309us) client_negotiation.cc:769] Sending connection context
1027 14:50:37.456539 (+ 407us) client_negotiation.cc:240] Negotiation successful
1027 14:50:37.456968 (+ 429us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":466,"thread_start_us":310,"threads_started":1}
I20251027 14:50:37.459540 798 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44309
I20251027 14:50:37.459869 798 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:50:37.460708 798 heartbeater.cc:507] Master 127.0.104.62:44309 requested a full tablet report, sending...
I20251027 14:50:37.462152 471 ts_manager.cc:194] Registered new tserver with Master: fe935882e7054faebbc9669dcc38f736 (127.0.104.2:42815)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:50:37.463071 471 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:51575
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:37.513320 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44309
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41987
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:37.838758 806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:37.839069 806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:37.839202 806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:37.847764 806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:37.847883 806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:37.848064 806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:50:37.858830 806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41987
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44309
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:37.860328 806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:37.861668 806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:37.870381 812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:37.870437 811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:37.870847 814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:37.871657 806 server_base.cc:1047] running on GCE node
I20251027 14:50:37.872162 806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:37.873057 806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:37.874280 806 hybrid_clock.cc:648] HybridClock initialized: now 1761576637874217 us; error 37 us; skew 500 ppm
I20251027 14:50:37.874570 806 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:37 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:37.878389 806 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:50:37.879953 806 webserver.cc:492] Webserver started at http://127.0.104.3:40981/ using document root <none> and password file <none>
I20251027 14:50:37.880714 806 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:37.880818 806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:37.881170 806 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:37.883765 806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "6915cf126c6e4b71bd59d3736cc32e77"
format_stamp: "Formatted at 2025-10-27 14:50:37 on dist-test-slave-hxxt"
server_key: "23cc71fdc0ca051c059ad07b3001d497"
server_key_iv: "f9f766de5796105ffceaa046c6e1deb4"
server_key_version: "encryptionkey@0"
I20251027 14:50:37.884570 806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "6915cf126c6e4b71bd59d3736cc32e77"
format_stamp: "Formatted at 2025-10-27 14:50:37 on dist-test-slave-hxxt"
server_key: "23cc71fdc0ca051c059ad07b3001d497"
server_key_iv: "f9f766de5796105ffceaa046c6e1deb4"
server_key_version: "encryptionkey@0"
I20251027 14:50:37.889513 806 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:50:37.892567 821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:37.893784 806 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251027 14:50:37.893921 806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "6915cf126c6e4b71bd59d3736cc32e77"
format_stamp: "Formatted at 2025-10-27 14:50:37 on dist-test-slave-hxxt"
server_key: "23cc71fdc0ca051c059ad07b3001d497"
server_key_iv: "f9f766de5796105ffceaa046c6e1deb4"
server_key_version: "encryptionkey@0"
I20251027 14:50:37.894106 806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:37.924271 806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:37.928148 806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:37.928409 806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:37.929298 806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:50:37.930711 806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:37.930799 806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:37.930898 806 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:37.930963 806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:37.948155 662 heartbeater.cc:499] Master 127.0.104.62:44309 was elected leader, sending a full tablet report...
I20251027 14:50:37.964425 806 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42915
I20251027 14:50:37.964479 933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42915 every 8 connection(s)
I20251027 14:50:37.965723 806 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:50:37.968086 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 806
I20251027 14:50:37.968281 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnListTablets.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:50:37.968765 416 external_mini_cluster.cc:1468] Setting key 09e65bd7eae02f362fb0fa511a2bfebd
Oct 27 14:50:37 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576637, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:37.983551 936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:37.968259 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:58763 (local address 127.0.104.62:44309)
1027 14:50:37.968706 (+ 447us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:37.968713 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:37.969826 (+ 1113us) server_negotiation.cc:408] Connection header received
1027 14:50:37.971843 (+ 2017us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:37.971850 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:37.971936 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:37.972037 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:37.973741 (+ 1704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.974677 (+ 936us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:37.975759 (+ 1082us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.976162 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:37.978751 (+ 2589us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:37.978787 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:37.978791 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:37.978820 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:37.981047 (+ 2227us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:37.981643 (+ 596us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:37.981646 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:37.981649 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:37.981716 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:37.981972 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:37.981975 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:37.981977 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:37.982166 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:37.982269 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:37.983048 (+ 779us) server_negotiation.cc:300] Negotiation successful
1027 14:50:37.983245 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":334,"thread_start_us":231,"threads_started":1}
I20251027 14:50:37.984596 937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:37.968889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44309 (local address 127.0.104.3:58763)
1027 14:50:37.969653 (+ 764us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:37.969705 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:37.971548 (+ 1843us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:37.972313 (+ 765us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:37.972322 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:37.972912 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:37.973570 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:37.973587 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.974820 (+ 1233us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:37.974824 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:37.975628 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:37.975639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:37.975889 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:37.976599 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:37.976623 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:37.978568 (+ 1945us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:37.981173 (+ 2605us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:37.981180 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:37.981195 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:37.981554 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:37.981796 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:37.981799 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:37.981802 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:37.981900 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:37.982255 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:37.982260 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:37.982574 (+ 314us) client_negotiation.cc:769] Sending connection context
1027 14:50:37.982940 (+ 366us) client_negotiation.cc:240] Negotiation successful
1027 14:50:37.983351 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":326,"threads_started":1}
I20251027 14:50:37.985769 934 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44309
I20251027 14:50:37.986101 934 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:37.986861 934 heartbeater.cc:507] Master 127.0.104.62:44309 requested a full tablet report, sending...
I20251027 14:50:37.988175 471 ts_manager.cc:194] Registered new tserver with Master: 6915cf126c6e4b71bd59d3736cc32e77 (127.0.104.3:42915)
I20251027 14:50:37.988965 471 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:58763
I20251027 14:50:37.997069 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576638, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:38.037268 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:38.023374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37072 (local address 127.0.104.1:37029)
1027 14:50:38.024059 (+ 685us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:38.024066 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:38.024093 (+ 27us) server_negotiation.cc:408] Connection header received
1027 14:50:38.024155 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:38.024161 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:38.024364 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:38.024562 (+ 198us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:38.026020 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.027220 (+ 1200us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:38.028337 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.028713 (+ 376us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:38.032188 (+ 3475us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:38.032260 (+ 72us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:38.032275 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:38.032308 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:38.034803 (+ 2495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.035505 (+ 702us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.035515 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.035519 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.035616 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.035938 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.035943 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.035946 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.036323 (+ 377us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:38.036611 (+ 288us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:38.036800 (+ 189us) server_negotiation.cc:300] Negotiation successful
1027 14:50:38.037006 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":555,"thread_start_us":452,"threads_started":1}
W20251027 14:50:38.038455 577 server_base.cc:1129] Unauthorized access attempt to method kudu.tserver.TabletServerService.ListTablets from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:37072
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[423](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:38.077457 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:38.065458 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37074 (local address 127.0.104.1:37029)
1027 14:50:38.065677 (+ 219us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:38.065684 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:38.065839 (+ 155us) server_negotiation.cc:408] Connection header received
1027 14:50:38.065978 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:38.065983 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:38.066044 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:38.066127 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:38.067507 (+ 1380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.068412 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:38.069491 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.069824 (+ 333us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:38.073256 (+ 3432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:38.073293 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:38.073297 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:38.073328 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:38.075588 (+ 2260us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.076086 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.076090 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.076092 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.076155 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.076359 (+ 204us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.076361 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.076363 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.076605 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:38.076776 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:38.076969 (+ 193us) server_negotiation.cc:300] Negotiation successful
1027 14:50:38.077153 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20251027 14:50:38.078791 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 534
I20251027 14:50:38.089804 942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:38.079709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37082 (local address 127.0.104.1:37029)
1027 14:50:38.079855 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:38.079860 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:38.080109 (+ 249us) server_negotiation.cc:408] Connection header received
1027 14:50:38.080239 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:38.080243 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:38.080292 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:38.080360 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:38.081644 (+ 1284us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.082434 (+ 790us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:38.083526 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.083931 (+ 405us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:38.085412 (+ 1481us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:38.085447 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:38.085451 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:38.085476 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:38.087600 (+ 2124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.088141 (+ 541us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.088146 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.088148 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.088210 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.088547 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.088553 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.088556 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.088788 (+ 232us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:38.088989 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:38.089283 (+ 294us) server_negotiation.cc:300] Negotiation successful
1027 14:50:38.089500 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:50:38.090481 657 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:38.172911 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 534
I20251027 14:50:38.186566 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 670
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:50:38.200497 949 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:38.187386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48984 (local address 127.0.104.2:42815)
1027 14:50:38.187852 (+ 466us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:38.187860 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:38.187885 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:50:38.187945 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:38.187952 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:38.188159 (+ 207us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:38.188345 (+ 186us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:38.189642 (+ 1297us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.190789 (+ 1147us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:38.191848 (+ 1059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.192234 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:38.195335 (+ 3101us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:38.195405 (+ 70us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:38.195420 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:38.195459 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:38.198265 (+ 2806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.198900 (+ 635us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.198908 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.198910 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.198989 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.199218 (+ 229us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.199221 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.199223 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.199495 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:38.199726 (+ 231us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:38.199900 (+ 174us) server_negotiation.cc:300] Negotiation successful
1027 14:50:38.200119 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":231,"threads_started":1}
I20251027 14:50:38.201714 793 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:38.285425 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 670
I20251027 14:50:38.304105 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 806
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[423](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576638, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:50:38.318711 951 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:38.305035 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57832 (local address 127.0.104.3:42915)
1027 14:50:38.305600 (+ 565us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:38.305607 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:38.305633 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:50:38.305701 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:38.305708 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:38.305910 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:38.306075 (+ 165us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:38.307469 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.308807 (+ 1338us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:38.309940 (+ 1133us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:38.310394 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:38.313592 (+ 3198us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:38.313655 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:38.313678 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:38.313711 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:38.316623 (+ 2912us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.317206 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.317214 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.317216 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.317297 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:38.317505 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:38.317507 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:38.317509 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:38.317782 (+ 273us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:38.317989 (+ 207us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:38.318188 (+ 199us) server_negotiation.cc:300] Negotiation successful
1027 14:50:38.318422 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":435,"thread_start_us":305,"threads_started":1}
I20251027 14:50:38.319918 929 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:38.401158 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 806
I20251027 14:50:38.416293 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 439
I20251027 14:50:38.417423 501 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:38.516163 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 439
2025-10-27T14:50:38Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnListTablets (4749 ms)
[ RUN ] SecurityITest.TestAuthorizationOnChecksum
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[956](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[956](info): set up 2 sockets
Oct 27 14:50:38 dist-test-slave-hxxt krb5kdc[956](info): commencing operation
krb5kdc: starting...
W20251027 14:50:40.372213 416 mini_kdc.cc:121] Time spent starting KDC: real 1.819s user 0.010s sys 0.004s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:50:40 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:50:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:50:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:40.567799 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43425
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43425
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:40.881666 972 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:40.882009 972 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:40.882146 972 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:40.890338 972 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:50:40.890468 972 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:40.890554 972 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:40.890615 972 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:50:40.890704 972 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:50:40.901141 972 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43425
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43425
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:40.902740 972 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:40.904088 972 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:40.911792 978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:40.911859 977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:40.912422 980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:40.913441 972 server_base.cc:1047] running on GCE node
I20251027 14:50:40.914170 972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:40.915632 972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:40.916831 972 hybrid_clock.cc:648] HybridClock initialized: now 1761576640916778 us; error 28 us; skew 500 ppm
I20251027 14:50:40.917155 972 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:40 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576640, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:40.920065 972 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:50:40.921536 972 webserver.cc:492] Webserver started at http://127.0.104.62:42029/ using document root <none> and password file <none>
I20251027 14:50:40.922230 972 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:40.922307 972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:40.922621 972 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:40.925064 972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "739d15febf07445f92069a04b2bdb590"
format_stamp: "Formatted at 2025-10-27 14:50:40 on dist-test-slave-hxxt"
server_key: "2610af95bb4b59e826ea5bbd32408248"
server_key_iv: "169872b1b55892b6882d9c99870bf66f"
server_key_version: "encryptionkey@0"
I20251027 14:50:40.925738 972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "739d15febf07445f92069a04b2bdb590"
format_stamp: "Formatted at 2025-10-27 14:50:40 on dist-test-slave-hxxt"
server_key: "2610af95bb4b59e826ea5bbd32408248"
server_key_iv: "169872b1b55892b6882d9c99870bf66f"
server_key_version: "encryptionkey@0"
I20251027 14:50:40.930613 972 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:50:40.933763 987 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:40.934993 972 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:50:40.935158 972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "739d15febf07445f92069a04b2bdb590"
format_stamp: "Formatted at 2025-10-27 14:50:40 on dist-test-slave-hxxt"
server_key: "2610af95bb4b59e826ea5bbd32408248"
server_key_iv: "169872b1b55892b6882d9c99870bf66f"
server_key_version: "encryptionkey@0"
I20251027 14:50:40.935365 972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:40.948078 972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:40.951620 972 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:40.951848 972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:40.968725 1038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43425 every 8 connection(s)
I20251027 14:50:40.968734 972 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43425
I20251027 14:50:40.970134 972 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:50:40.971648 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 972
I20251027 14:50:40.971838 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:50:40.972361 416 external_mini_cluster.cc:1468] Setting key 0c3a85bf916173c20cc07197186aa862
I20251027 14:50:40.974421 1039 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 27 14:50:40 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:40.984659 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590: Bootstrap starting.
I20251027 14:50:40.988032 1039 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:40.989289 1039 log.cc:826] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:40.991621 1042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:40.974231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49044 (local address 127.0.104.62:43425)
1027 14:50:40.974973 (+ 742us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:40.974984 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:40.975027 (+ 43us) server_negotiation.cc:408] Connection header received
1027 14:50:40.976311 (+ 1284us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:40.976335 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:40.976774 (+ 439us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:40.977162 (+ 388us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:40.978715 (+ 1553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:40.980083 (+ 1368us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:40.981215 (+ 1132us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:40.981730 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:40.984965 (+ 3235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:40.985056 (+ 91us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:40.985071 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:40.985105 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:40.987878 (+ 2773us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:40.988671 (+ 793us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:40.988677 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:40.988684 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:40.988775 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:40.989081 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:40.989084 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:40.989085 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:40.989563 (+ 478us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:40.989839 (+ 276us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:40.990353 (+ 514us) server_negotiation.cc:300] Negotiation successful
1027 14:50:40.990721 (+ 368us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":347,"threads_started":1}
I20251027 14:50:40.992234 1039 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590: No bootstrap required, opened a new log
I20251027 14:50:40.997710 1039 raft_consensus.cc:359] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739d15febf07445f92069a04b2bdb590" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43425 } }
I20251027 14:50:40.998024 1039 raft_consensus.cc:385] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:40.998128 1039 raft_consensus.cc:740] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 739d15febf07445f92069a04b2bdb590, State: Initialized, Role: FOLLOWER
I20251027 14:50:40.998716 1039 consensus_queue.cc:260] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739d15febf07445f92069a04b2bdb590" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43425 } }
I20251027 14:50:40.998916 1039 raft_consensus.cc:399] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:50:40.999029 1039 raft_consensus.cc:493] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:50:40.999171 1039 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:41.001336 1039 raft_consensus.cc:515] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739d15febf07445f92069a04b2bdb590" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43425 } }
I20251027 14:50:41.001804 1039 leader_election.cc:304] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 739d15febf07445f92069a04b2bdb590; no voters:
I20251027 14:50:41.002394 1039 leader_election.cc:290] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:50:41.002545 1044 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:41.003132 1044 raft_consensus.cc:697] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [term 1 LEADER]: Becoming Leader. State: Replica: 739d15febf07445f92069a04b2bdb590, State: Running, Role: LEADER
I20251027 14:50:41.003602 1044 consensus_queue.cc:237] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739d15febf07445f92069a04b2bdb590" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43425 } }
I20251027 14:50:41.004231 1039 sys_catalog.cc:565] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:50:41.006357 1046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 739d15febf07445f92069a04b2bdb590. Latest consensus state: current_term: 1 leader_uuid: "739d15febf07445f92069a04b2bdb590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739d15febf07445f92069a04b2bdb590" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43425 } } }
I20251027 14:50:41.006610 1046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:41.007012 1045 sys_catalog.cc:455] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "739d15febf07445f92069a04b2bdb590" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "739d15febf07445f92069a04b2bdb590" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43425 } } }
I20251027 14:50:41.007246 1045 sys_catalog.cc:458] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590 [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:41.009255 1050 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:50:41.013836 1050 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:50:41.021697 1050 catalog_manager.cc:1357] Generated new cluster ID: 625f3eae4eb242deafce19f9dfd940be
I20251027 14:50:41.021806 1050 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:50:41.042966 1050 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:50:41.044692 1050 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:50:41.059621 1050 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 739d15febf07445f92069a04b2bdb590: Generated new TSK 0
I20251027 14:50:41.060817 1050 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:41.135581 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43425
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:41.447781 1067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:41.448073 1067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:41.448210 1067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:41.456274 1067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:41.456364 1067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:41.456568 1067 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:50:41.466565 1067 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:43425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:41.468061 1067 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:41.469435 1067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:41.478256 1072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:41.478282 1073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:41.478765 1075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:41.479146 1067 server_base.cc:1047] running on GCE node
I20251027 14:50:41.479744 1067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:41.480670 1067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:41.481877 1067 hybrid_clock.cc:648] HybridClock initialized: now 1761576641481808 us; error 48 us; skew 500 ppm
I20251027 14:50:41.482136 1067 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:41 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:41.485286 1067 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:50:41.486706 1067 webserver.cc:492] Webserver started at http://127.0.104.1:36665/ using document root <none> and password file <none>
I20251027 14:50:41.487421 1067 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:41.487514 1067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:41.487849 1067 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:41.490453 1067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "1373506722774d719ee901b1485f4739"
format_stamp: "Formatted at 2025-10-27 14:50:41 on dist-test-slave-hxxt"
server_key: "f5221f1ef7c200de8c264b3f0948cfc8"
server_key_iv: "fdce026e7d99b5f4996ff5819c83d421"
server_key_version: "encryptionkey@0"
I20251027 14:50:41.491297 1067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "1373506722774d719ee901b1485f4739"
format_stamp: "Formatted at 2025-10-27 14:50:41 on dist-test-slave-hxxt"
server_key: "f5221f1ef7c200de8c264b3f0948cfc8"
server_key_iv: "fdce026e7d99b5f4996ff5819c83d421"
server_key_version: "encryptionkey@0"
I20251027 14:50:41.496400 1067 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:50:41.499653 1082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:41.500988 1067 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:50:41.501127 1067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "1373506722774d719ee901b1485f4739"
format_stamp: "Formatted at 2025-10-27 14:50:41 on dist-test-slave-hxxt"
server_key: "f5221f1ef7c200de8c264b3f0948cfc8"
server_key_iv: "fdce026e7d99b5f4996ff5819c83d421"
server_key_version: "encryptionkey@0"
I20251027 14:50:41.501325 1067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:41.528903 1067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:41.532855 1067 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:41.533109 1067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:41.534045 1067 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:50:41.535496 1067 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:41.535589 1067 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:41.535687 1067 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:41.535753 1067 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:41.565858 1067 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45717
I20251027 14:50:41.565904 1194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45717 every 8 connection(s)
I20251027 14:50:41.567180 1067 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:50:41.571117 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1067
I20251027 14:50:41.571359 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:50:41.571950 416 external_mini_cluster.cc:1468] Setting key df083534dde82af4a60c61152362e5e2
Oct 27 14:50:41 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:41.585887 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:41.569821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:51855 (local address 127.0.104.62:43425)
1027 14:50:41.570225 (+ 404us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:41.570230 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:41.571162 (+ 932us) server_negotiation.cc:408] Connection header received
1027 14:50:41.572848 (+ 1686us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:41.572855 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:41.572943 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:41.573046 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:41.574860 (+ 1814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:41.575751 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:41.576870 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:41.577299 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:41.580035 (+ 2736us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:41.580069 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:41.580075 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:41.580111 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:41.583012 (+ 2901us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:41.583732 (+ 720us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:41.583738 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:41.583742 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:41.583830 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:41.584118 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:41.584123 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:41.584126 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:41.584447 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:41.584604 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:41.585286 (+ 682us) server_negotiation.cc:300] Negotiation successful
1027 14:50:41.585536 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":211,"threads_started":1}
I20251027 14:50:41.586668 1198 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:41.570313 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43425 (local address 127.0.104.1:51855)
1027 14:50:41.571003 (+ 690us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:41.571040 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:41.572512 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:41.573329 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:41.573338 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:41.574002 (+ 664us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:41.574682 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:41.574700 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:41.575896 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:41.575900 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:41.576733 (+ 833us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:41.576744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:41.577009 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:41.577806 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:41.577830 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:41.579840 (+ 2010us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:41.583151 (+ 3311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:41.583158 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:41.583174 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:41.583622 (+ 448us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:41.583924 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:41.583927 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:41.583929 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:41.584026 (+ 97us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:41.584560 (+ 534us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:41.584566 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:41.584862 (+ 296us) client_negotiation.cc:769] Sending connection context
1027 14:50:41.585176 (+ 314us) client_negotiation.cc:240] Negotiation successful
1027 14:50:41.585536 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":321,"threads_started":1}
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:50:41.588294 1195 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43425
I20251027 14:50:41.588649 1195 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:41.589394 1195 heartbeater.cc:507] Master 127.0.104.62:43425 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:50:41.591884 1004 ts_manager.cc:194] Registered new tserver with Master: 1373506722774d719ee901b1485f4739 (127.0.104.1:45717)
I20251027 14:50:41.593868 1004 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:51855
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:41.640290 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43425
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:41.952090 1203 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:41.952401 1203 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:41.952553 1203 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:41.960556 1203 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:41.960649 1203 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:41.960805 1203 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:50:41.970862 1203 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:43425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:41.972332 1203 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:41.973666 1203 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:41.982632 1208 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:41.983461 1209 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:41.983855 1211 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:41.984807 1203 server_base.cc:1047] running on GCE node
I20251027 14:50:41.985384 1203 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:41.986243 1203 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:41.987454 1203 hybrid_clock.cc:648] HybridClock initialized: now 1761576641987383 us; error 41 us; skew 500 ppm
I20251027 14:50:41.987722 1203 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:41 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:41.990775 1203 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:50:41.992269 1203 webserver.cc:492] Webserver started at http://127.0.104.2:38173/ using document root <none> and password file <none>
I20251027 14:50:41.992990 1203 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:41.993080 1203 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:41.993423 1203 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:41.995946 1203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "eb08c6feb2884775b972b61e597776f4"
format_stamp: "Formatted at 2025-10-27 14:50:41 on dist-test-slave-hxxt"
server_key: "cec47b63b2f934a548b188489f95e073"
server_key_iv: "945d8b04ed46da56af60365969c487b6"
server_key_version: "encryptionkey@0"
I20251027 14:50:41.996757 1203 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "eb08c6feb2884775b972b61e597776f4"
format_stamp: "Formatted at 2025-10-27 14:50:41 on dist-test-slave-hxxt"
server_key: "cec47b63b2f934a548b188489f95e073"
server_key_iv: "945d8b04ed46da56af60365969c487b6"
server_key_version: "encryptionkey@0"
I20251027 14:50:42.002038 1203 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:50:42.005689 1218 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:42.007062 1203 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:50:42.007220 1203 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "eb08c6feb2884775b972b61e597776f4"
format_stamp: "Formatted at 2025-10-27 14:50:41 on dist-test-slave-hxxt"
server_key: "cec47b63b2f934a548b188489f95e073"
server_key_iv: "945d8b04ed46da56af60365969c487b6"
server_key_version: "encryptionkey@0"
I20251027 14:50:42.007440 1203 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:42.026494 1203 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:42.030496 1203 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:42.030764 1203 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:42.031728 1203 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:50:42.033247 1203 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:42.033345 1203 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:42.033452 1203 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:42.033519 1203 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:42.063647 1203 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41173
I20251027 14:50:42.063684 1330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41173 every 8 connection(s)
I20251027 14:50:42.064990 1203 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:50:42.065369 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1203
I20251027 14:50:42.065569 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:50:42.066023 416 external_mini_cluster.cc:1468] Setting key e4ee514998d31e8f629ba262b5bfca59
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576641, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:50:42.084721 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.067667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:43313 (local address 127.0.104.62:43425)
1027 14:50:42.067822 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.067828 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.069205 (+ 1377us) server_negotiation.cc:408] Connection header received
1027 14:50:42.071182 (+ 1977us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.071187 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.071253 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.071337 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:42.073325 (+ 1988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.074573 (+ 1248us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.075618 (+ 1045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.076017 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.078844 (+ 2827us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:42.078878 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:42.078882 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:42.078912 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:42.081993 (+ 3081us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.082688 (+ 695us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.082695 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.082699 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.082801 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.083120 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.083124 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.083126 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.083368 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:42.083481 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.084205 (+ 724us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.084445 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:50:42.085781 1333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.068212 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43425 (local address 127.0.104.2:43313)
1027 14:50:42.069012 (+ 800us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.069060 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.070857 (+ 1797us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.071588 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.071600 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.072312 (+ 712us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:42.073155 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.073175 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.074703 (+ 1528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.074707 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.075495 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.075505 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.075699 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.076339 (+ 640us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:42.076364 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:42.078644 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:42.082136 (+ 3492us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:42.082142 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:42.082157 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:42.082558 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:42.082902 (+ 344us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:42.082905 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:42.082908 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:42.083010 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:42.083460 (+ 450us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:42.083466 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:42.083833 (+ 367us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.084125 (+ 292us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.084658 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":372,"threads_started":1}
I20251027 14:50:42.087221 1331 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43425
I20251027 14:50:42.087597 1331 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:42.088366 1331 heartbeater.cc:507] Master 127.0.104.62:43425 requested a full tablet report, sending...
I20251027 14:50:42.089902 1004 ts_manager.cc:194] Registered new tserver with Master: eb08c6feb2884775b972b61e597776f4 (127.0.104.2:41173)
I20251027 14:50:42.090934 1004 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:43313
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:42.135953 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43425
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41603
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_enforce_access_control with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:42.448097 1338 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:42.448397 1338 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:42.448529 1338 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:42.456492 1338 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:42.456588 1338 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:42.456734 1338 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:50:42.466936 1338 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41603
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:43425
--tserver_enforce_access_control=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:42.468411 1338 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:42.469617 1338 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:42.478323 1344 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:42.478348 1343 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:42.479249 1346 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:42.479513 1338 server_base.cc:1047] running on GCE node
I20251027 14:50:42.480053 1338 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:42.480962 1338 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:42.482174 1338 hybrid_clock.cc:648] HybridClock initialized: now 1761576642482110 us; error 36 us; skew 500 ppm
I20251027 14:50:42.482441 1338 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:42.485491 1338 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:50:42.486927 1338 webserver.cc:492] Webserver started at http://127.0.104.3:41617/ using document root <none> and password file <none>
I20251027 14:50:42.487655 1338 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:42.487756 1338 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:42.488123 1338 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:42.490671 1338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "ef2d9b5dfcff4fa6ae74d9c919926588"
format_stamp: "Formatted at 2025-10-27 14:50:42 on dist-test-slave-hxxt"
server_key: "e1523d2e1b61183d9a6cc5093aa4e58d"
server_key_iv: "0409cacce3ecf1b3076cb3259d69be19"
server_key_version: "encryptionkey@0"
I20251027 14:50:42.491466 1338 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "ef2d9b5dfcff4fa6ae74d9c919926588"
format_stamp: "Formatted at 2025-10-27 14:50:42 on dist-test-slave-hxxt"
server_key: "e1523d2e1b61183d9a6cc5093aa4e58d"
server_key_iv: "0409cacce3ecf1b3076cb3259d69be19"
server_key_version: "encryptionkey@0"
I20251027 14:50:42.496939 1338 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251027 14:50:42.500310 1353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:42.501703 1338 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:50:42.501861 1338 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "ef2d9b5dfcff4fa6ae74d9c919926588"
format_stamp: "Formatted at 2025-10-27 14:50:42 on dist-test-slave-hxxt"
server_key: "e1523d2e1b61183d9a6cc5093aa4e58d"
server_key_iv: "0409cacce3ecf1b3076cb3259d69be19"
server_key_version: "encryptionkey@0"
I20251027 14:50:42.502063 1338 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:42.524202 1338 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:42.527969 1338 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:42.528229 1338 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:42.529237 1338 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:50:42.530689 1338 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:42.530786 1338 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:42.530891 1338 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:42.530961 1338 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:42.560370 1338 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36589
I20251027 14:50:42.560449 1465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36589 every 8 connection(s)
I20251027 14:50:42.561754 1338 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:50:42.571834 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1338
I20251027 14:50:42.572021 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizationOnChecksum.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:50:42.572448 416 external_mini_cluster.cc:1468] Setting key cb781704314b3217b046ef23108ecfa7
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:42.580183 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.564177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:40173 (local address 127.0.104.62:43425)
1027 14:50:42.564331 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.564335 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.565522 (+ 1187us) server_negotiation.cc:408] Connection header received
1027 14:50:42.567139 (+ 1617us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.567147 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.567230 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.567343 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:42.568920 (+ 1577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.570113 (+ 1193us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.571130 (+ 1017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.571412 (+ 282us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.574389 (+ 2977us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:42.574428 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:42.574432 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:42.574462 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:42.577316 (+ 2854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.577952 (+ 636us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.577956 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.577959 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.578022 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.578380 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.578384 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.578386 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.578630 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:42.578804 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.579591 (+ 787us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.579830 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:50:42.581138 1468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.564662 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43425 (local address 127.0.104.3:40173)
1027 14:50:42.565347 (+ 685us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.565390 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.566852 (+ 1462us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.567534 (+ 682us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.567541 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.568056 (+ 515us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:42.568754 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.568772 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.570245 (+ 1473us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.570249 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.571026 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.571036 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.571232 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.571917 (+ 685us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:42.571941 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:42.574208 (+ 2267us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:42.577448 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:42.577457 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:42.577475 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:42.577834 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:42.578113 (+ 279us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:42.578118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:42.578120 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:42.578225 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:42.578785 (+ 560us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:42.578794 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:42.579148 (+ 354us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.579475 (+ 327us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.579962 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":445,"thread_start_us":295,"threads_started":1}
I20251027 14:50:42.582581 1466 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43425
I20251027 14:50:42.582935 1466 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:42.583724 1466 heartbeater.cc:507] Master 127.0.104.62:43425 requested a full tablet report, sending...
I20251027 14:50:42.585246 1004 ts_manager.cc:194] Registered new tserver with Master: ef2d9b5dfcff4fa6ae74d9c919926588 (127.0.104.3:36589)
I20251027 14:50:42.586025 1004 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:40173
I20251027 14:50:42.587725 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:50:42.596613 1195 heartbeater.cc:499] Master 127.0.104.62:43425 was elected leader, sending a full tablet report...
I20251027 14:50:42.606343 1197 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.594197 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49050 (local address 127.0.104.62:43425)
1027 14:50:42.594404 (+ 207us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.594411 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.594822 (+ 411us) server_negotiation.cc:408] Connection header received
1027 14:50:42.594955 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.594959 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.595024 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.595111 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:42.596743 (+ 1632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.598421 (+ 1678us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.599605 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.599911 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.601633 (+ 1722us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:42.601671 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:42.601674 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:42.601704 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:42.604029 (+ 2325us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.604674 (+ 645us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.604679 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.604681 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.604747 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.605132 (+ 385us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.605135 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.605137 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.605389 (+ 252us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:42.605606 (+ 217us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.605845 (+ 239us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.606038 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251027 14:50:42.612851 1004 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:49050:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:50:42.617373 1004 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:42.639489 1480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.630210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:41173 (local address 127.0.0.1:42214)
1027 14:50:42.631830 (+ 1620us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.631844 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.631941 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.633742 (+ 1801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.633746 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.633772 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:42.634520 (+ 748us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.634533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.636955 (+ 2422us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.636964 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.638526 (+ 1562us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.638538 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.638721 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.638789 (+ 68us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.638939 (+ 150us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.639135 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1499,"thread_start_us":1350,"threads_started":1}
I20251027 14:50:42.640358 1482 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.631937 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45717 (local address 127.0.0.1:42412)
1027 14:50:42.633352 (+ 1415us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.633371 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.633478 (+ 107us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.634163 (+ 685us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.634170 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.634201 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:42.635404 (+ 1203us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.635418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.637627 (+ 2209us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.637634 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.639359 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.639373 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.639544 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.639570 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.639631 (+ 61us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.639745 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1287,"thread_start_us":1155,"threads_started":1}
I20251027 14:50:42.641022 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.630734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42214 (local address 127.0.104.2:41173)
1027 14:50:42.631404 (+ 670us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.631413 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.632029 (+ 616us) server_negotiation.cc:408] Connection header received
1027 14:50:42.632151 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.632159 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.632439 (+ 280us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.633707 (+ 1268us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:42.634729 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.636675 (+ 1946us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.638661 (+ 1986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.639684 (+ 1023us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.639843 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.639980 (+ 137us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.640147 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":428,"threads_started":1}
I20251027 14:50:42.641142 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.629657 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55806 (local address 127.0.104.3:36589)
1027 14:50:42.630951 (+ 1294us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.630961 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.631472 (+ 511us) server_negotiation.cc:408] Connection header received
1027 14:50:42.632133 (+ 661us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.632144 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.632469 (+ 325us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.632686 (+ 217us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:42.635189 (+ 2503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.636635 (+ 1446us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.638987 (+ 2352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.640118 (+ 1131us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.640264 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.640697 (+ 433us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.640876 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1135,"thread_start_us":1010,"threads_started":1}
I20251027 14:50:42.641705 1478 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.629480 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36589 (local address 127.0.0.1:55806)
1027 14:50:42.631328 (+ 1848us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.631367 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.631646 (+ 279us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.633609 (+ 1963us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.633625 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.633656 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:42.634520 (+ 864us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.634533 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.636912 (+ 2379us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.636918 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.638806 (+ 1888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.638822 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.640455 (+ 1633us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.640481 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.641315 (+ 834us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.641431 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1588,"thread_start_us":1483,"threads_started":1}
I20251027 14:50:42.642931 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.632226 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42412 (local address 127.0.104.1:45717)
1027 14:50:42.633616 (+ 1390us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.633625 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.633651 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:50:42.633754 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.633763 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.633979 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.634157 (+ 178us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:42.635607 (+ 1450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.637408 (+ 1801us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.639997 (+ 2589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.642258 (+ 2261us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.642400 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.642525 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.642687 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1244,"thread_start_us":1146,"threads_started":1}
I20251027 14:50:42.645556 1266 tablet_service.cc:1505] Processing CreateTablet for tablet 756183ae789a484aa7225afe8213ff4c (DEFAULT_TABLE table=test-table [id=3ecf07c783114931a9bc9dc6b22d4f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:42.645829 1401 tablet_service.cc:1505] Processing CreateTablet for tablet 756183ae789a484aa7225afe8213ff4c (DEFAULT_TABLE table=test-table [id=3ecf07c783114931a9bc9dc6b22d4f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:42.647089 1266 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756183ae789a484aa7225afe8213ff4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:42.647274 1401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756183ae789a484aa7225afe8213ff4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:42.646854 1130 tablet_service.cc:1505] Processing CreateTablet for tablet 756183ae789a484aa7225afe8213ff4c (DEFAULT_TABLE table=test-table [id=3ecf07c783114931a9bc9dc6b22d4f04]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:42.648164 1130 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 756183ae789a484aa7225afe8213ff4c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:42.657294 1484 tablet_bootstrap.cc:492] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: Bootstrap starting.
I20251027 14:50:42.661127 1485 tablet_bootstrap.cc:492] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: Bootstrap starting.
I20251027 14:50:42.661823 1484 tablet_bootstrap.cc:654] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:42.663157 1484 log.cc:826] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:42.664623 1485 tablet_bootstrap.cc:654] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:42.665990 1485 log.cc:826] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:42.667450 1486 tablet_bootstrap.cc:492] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: Bootstrap starting.
I20251027 14:50:42.671018 1486 tablet_bootstrap.cc:654] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:42.671980 1484 tablet_bootstrap.cc:492] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: No bootstrap required, opened a new log
I20251027 14:50:42.672248 1484 ts_tablet_manager.cc:1403] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: Time spent bootstrapping tablet: real 0.015s user 0.003s sys 0.009s
I20251027 14:50:42.672269 1486 log.cc:826] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:42.672451 1485 tablet_bootstrap.cc:492] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: No bootstrap required, opened a new log
I20251027 14:50:42.672700 1485 ts_tablet_manager.cc:1403] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: Time spent bootstrapping tablet: real 0.012s user 0.010s sys 0.000s
I20251027 14:50:42.675352 1486 tablet_bootstrap.cc:492] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: No bootstrap required, opened a new log
I20251027 14:50:42.675594 1486 ts_tablet_manager.cc:1403] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20251027 14:50:42.678174 1484 raft_consensus.cc:359] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.678547 1484 raft_consensus.cc:385] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:42.678649 1484 raft_consensus.cc:740] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb08c6feb2884775b972b61e597776f4, State: Initialized, Role: FOLLOWER
I20251027 14:50:42.679211 1485 raft_consensus.cc:359] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.679440 1484 consensus_queue.cc:260] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.679574 1485 raft_consensus.cc:385] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:42.679656 1485 raft_consensus.cc:740] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ef2d9b5dfcff4fa6ae74d9c919926588, State: Initialized, Role: FOLLOWER
I20251027 14:50:42.680253 1485 consensus_queue.cc:260] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.681257 1331 heartbeater.cc:499] Master 127.0.104.62:43425 was elected leader, sending a full tablet report...
I20251027 14:50:42.681846 1484 ts_tablet_manager.cc:1434] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: Time spent starting tablet: real 0.009s user 0.005s sys 0.005s
I20251027 14:50:42.682345 1486 raft_consensus.cc:359] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.682798 1486 raft_consensus.cc:385] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:42.682925 1486 raft_consensus.cc:740] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1373506722774d719ee901b1485f4739, State: Initialized, Role: FOLLOWER
I20251027 14:50:42.683553 1486 consensus_queue.cc:260] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.683845 1485 ts_tablet_manager.cc:1434] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
I20251027 14:50:42.684285 1466 heartbeater.cc:499] Master 127.0.104.62:43425 was elected leader, sending a full tablet report...
I20251027 14:50:42.687315 1486 ts_tablet_manager.cc:1434] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20251027 14:50:42.813294 1467 tablet.cc:2378] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:50:42.817092 1332 tablet.cc:2378] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:50:42.819298 1196 tablet.cc:2378] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:50:42.864840 1490 raft_consensus.cc:493] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:42.865142 1490 raft_consensus.cc:515] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.866925 1490 leader_election.cc:290] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ef2d9b5dfcff4fa6ae74d9c919926588 (127.0.104.3:36589), 1373506722774d719ee901b1485f4739 (127.0.104.1:45717)
I20251027 14:50:42.873749 1494 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.867891 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45717 (local address 127.0.104.2:55721)
1027 14:50:42.868494 (+ 603us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.868512 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.868638 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.868911 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.868916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.868944 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:42.869511 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.869519 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.871314 (+ 1795us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.871321 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.873004 (+ 1683us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.873019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.873229 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.873247 (+ 18us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.873306 (+ 59us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.873428 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":349,"threads_started":1}
I20251027 14:50:42.873772 1493 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.867225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36589 (local address 127.0.104.2:53589)
1027 14:50:42.867752 (+ 527us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:42.867768 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:42.867897 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:42.868308 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:42.868314 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:42.868347 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:42.869107 (+ 760us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.869119 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.870591 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.870595 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:42.873004 (+ 2409us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:42.873019 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.873202 (+ 183us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.873234 (+ 32us) client_negotiation.cc:769] Sending connection context
1027 14:50:42.873306 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:50:42.873428 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":428,"thread_start_us":296,"threads_started":1}
I20251027 14:50:42.874684 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.868049 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:55721 (local address 127.0.104.1:45717)
1027 14:50:42.868204 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.868209 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.868606 (+ 397us) server_negotiation.cc:408] Connection header received
1027 14:50:42.868748 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.868752 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.868807 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.868875 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:42.869641 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.870825 (+ 1184us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.873159 (+ 2334us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.874214 (+ 1055us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.874264 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.874349 (+ 85us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.874467 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:50:42.874701 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.867296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:53589 (local address 127.0.104.3:36589)
1027 14:50:42.867482 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.867490 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.867856 (+ 366us) server_negotiation.cc:408] Connection header received
1027 14:50:42.868052 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.868059 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.868149 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.868247 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:42.869246 (+ 999us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.870427 (+ 1181us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.873158 (+ 2731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.874214 (+ 1056us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.874264 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.874349 (+ 85us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.874467 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:50:42.875814 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "756183ae789a484aa7225afe8213ff4c" candidate_uuid: "eb08c6feb2884775b972b61e597776f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1373506722774d719ee901b1485f4739" is_pre_election: true
I20251027 14:50:42.875815 1421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "756183ae789a484aa7225afe8213ff4c" candidate_uuid: "eb08c6feb2884775b972b61e597776f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" is_pre_election: true
I20251027 14:50:42.876319 1150 raft_consensus.cc:2468] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb08c6feb2884775b972b61e597776f4 in term 0.
I20251027 14:50:42.876315 1421 raft_consensus.cc:2468] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate eb08c6feb2884775b972b61e597776f4 in term 0.
I20251027 14:50:42.876945 1221 leader_election.cc:304] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1373506722774d719ee901b1485f4739, eb08c6feb2884775b972b61e597776f4; no voters:
I20251027 14:50:42.877315 1490 raft_consensus.cc:2804] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:42.877445 1490 raft_consensus.cc:493] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:42.877543 1490 raft_consensus.cc:3060] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:42.879663 1490 raft_consensus.cc:515] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.880162 1490 leader_election.cc:290] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [CANDIDATE]: Term 1 election: Requested vote from peers ef2d9b5dfcff4fa6ae74d9c919926588 (127.0.104.3:36589), 1373506722774d719ee901b1485f4739 (127.0.104.1:45717)
I20251027 14:50:42.880599 1421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "756183ae789a484aa7225afe8213ff4c" candidate_uuid: "eb08c6feb2884775b972b61e597776f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588"
I20251027 14:50:42.880687 1150 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "756183ae789a484aa7225afe8213ff4c" candidate_uuid: "eb08c6feb2884775b972b61e597776f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1373506722774d719ee901b1485f4739"
I20251027 14:50:42.880761 1421 raft_consensus.cc:3060] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:42.880856 1150 raft_consensus.cc:3060] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:42.883045 1421 raft_consensus.cc:2468] T 756183ae789a484aa7225afe8213ff4c P ef2d9b5dfcff4fa6ae74d9c919926588 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb08c6feb2884775b972b61e597776f4 in term 1.
I20251027 14:50:42.883394 1221 leader_election.cc:304] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: eb08c6feb2884775b972b61e597776f4, ef2d9b5dfcff4fa6ae74d9c919926588; no voters:
I20251027 14:50:42.883618 1150 raft_consensus.cc:2468] T 756183ae789a484aa7225afe8213ff4c P 1373506722774d719ee901b1485f4739 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate eb08c6feb2884775b972b61e597776f4 in term 1.
I20251027 14:50:42.883669 1490 raft_consensus.cc:2804] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:42.884143 1490 raft_consensus.cc:697] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [term 1 LEADER]: Becoming Leader. State: Replica: eb08c6feb2884775b972b61e597776f4, State: Running, Role: LEADER
I20251027 14:50:42.884660 1490 consensus_queue.cc:237] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } }
I20251027 14:50:42.889070 1002 catalog_manager.cc:5649] T 756183ae789a484aa7225afe8213ff4c P eb08c6feb2884775b972b61e597776f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to eb08c6feb2884775b972b61e597776f4 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "eb08c6feb2884775b972b61e597776f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ef2d9b5dfcff4fa6ae74d9c919926588" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36589 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "1373506722774d719ee901b1485f4739" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45717 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "eb08c6feb2884775b972b61e597776f4" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41173 } health_report { overall_health: HEALTHY } } }
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576640, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:42.937649 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.924746 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42416 (local address 127.0.104.1:45717)
1027 14:50:42.924924 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.924931 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.925284 (+ 353us) server_negotiation.cc:408] Connection header received
1027 14:50:42.925438 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.925443 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.925516 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.925596 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:42.927626 (+ 2030us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.928505 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.929623 (+ 1118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.930019 (+ 396us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.933079 (+ 3060us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:42.933133 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:42.933148 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:42.933179 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:42.935664 (+ 2485us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.936170 (+ 506us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.936176 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.936179 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.936240 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.936507 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.936510 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.936512 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.936766 (+ 254us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:42.936971 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.937150 (+ 179us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.937327 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:42.979485 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:42.966464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42424 (local address 127.0.104.1:45717)
1027 14:50:42.966636 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:42.966644 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:42.966987 (+ 343us) server_negotiation.cc:408] Connection header received
1027 14:50:42.967178 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:42.967183 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:42.967245 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:42.967333 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:42.969317 (+ 1984us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.970151 (+ 834us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:42.971215 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:42.971662 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:42.974843 (+ 3181us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:42.974882 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:42.974886 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:42.974929 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:42.977456 (+ 2527us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.978134 (+ 678us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.978138 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.978140 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.978199 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:42.978455 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:42.978457 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:42.978459 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:42.978649 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:42.978824 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:42.979085 (+ 261us) server_negotiation.cc:300] Negotiation successful
1027 14:50:42.979250 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
Oct 27 14:50:42 dist-test-slave-hxxt krb5kdc[956](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 27 14:50:43 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:43.019205 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:43.007174 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42428 (local address 127.0.104.1:45717)
1027 14:50:43.007379 (+ 205us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:43.007385 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:43.007576 (+ 191us) server_negotiation.cc:408] Connection header received
1027 14:50:43.007749 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:43.007754 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:43.007814 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:43.007892 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:43.009456 (+ 1564us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.010318 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:43.011413 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.011828 (+ 415us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:43.014774 (+ 2946us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:43.014809 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:43.014813 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:43.014843 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:43.017122 (+ 2279us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.017658 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.017663 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.017665 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.017741 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.018004 (+ 263us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.018022 (+ 18us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.018024 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.018214 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:43.018416 (+ 202us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:43.018688 (+ 272us) server_negotiation.cc:300] Negotiation successful
1027 14:50:43.018911 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
W20251027 14:50:43.020881 1110 tablet_service.cc:3012] Rejecting scan request for tablet 756183ae789a484aa7225afe8213ff4c: Uninitialized: clean time has not yet been initialized
I20251027 14:50:43.023837 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1067
I20251027 14:50:43.034073 1483 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:43.024633 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42440 (local address 127.0.104.1:45717)
1027 14:50:43.024762 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:43.024766 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:43.025083 (+ 317us) server_negotiation.cc:408] Connection header received
1027 14:50:43.025199 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:43.025203 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:43.025267 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:43.025340 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:43.026557 (+ 1217us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.027366 (+ 809us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:43.028389 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.029163 (+ 774us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:43.030125 (+ 962us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:43.030152 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:43.030156 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:43.030177 (+ 21us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:43.032304 (+ 2127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.032827 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.032831 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.032833 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.032891 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.033099 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.033102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.033104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.033292 (+ 188us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:43.033442 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:43.033649 (+ 207us) server_negotiation.cc:300] Negotiation successful
1027 14:50:43.033832 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251027 14:50:43.034690 1190 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:43.122397 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1067
I20251027 14:50:43.135938 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1203
Oct 27 14:50:43 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:50:43.149159 1481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:43.136659 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42220 (local address 127.0.104.2:41173)
1027 14:50:43.136810 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:43.136815 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:43.136835 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:50:43.137033 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:43.137037 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:43.137102 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:43.137183 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:43.138513 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.139385 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:43.140571 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.140962 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:43.144135 (+ 3173us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:43.144172 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:43.144188 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:43.144220 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:43.146770 (+ 2550us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.147389 (+ 619us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.147396 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.147398 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.147487 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.147765 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.147771 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.147773 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.148039 (+ 266us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:43.148340 (+ 301us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:43.148646 (+ 306us) server_negotiation.cc:300] Negotiation successful
1027 14:50:43.148865 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:50:43.149770 1326 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:43.236495 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1203
I20251027 14:50:43.249943 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1338
Oct 27 14:50:43 dist-test-slave-hxxt krb5kdc[956](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576642, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:50:43.262969 1479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:43.250646 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55808 (local address 127.0.104.3:36589)
1027 14:50:43.250806 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:43.250812 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:43.250834 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:50:43.250985 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:43.251002 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:43.251066 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:43.251158 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:43.252489 (+ 1331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.253313 (+ 824us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:43.254404 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:43.254789 (+ 385us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:43.257996 (+ 3207us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:43.258035 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:43.258051 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:43.258090 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:43.260646 (+ 2556us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.261239 (+ 593us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.261247 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.261249 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.261331 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:43.261597 (+ 266us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:43.261602 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:43.261605 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:43.261875 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:43.262178 (+ 303us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:43.262427 (+ 249us) server_negotiation.cc:300] Negotiation successful
1027 14:50:43.262664 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:50:43.263607 1461 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:43.348296 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1338
I20251027 14:50:43.361218 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 972
I20251027 14:50:43.362320 1034 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:43.464869 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 972
2025-10-27T14:50:43Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizationOnChecksum (4949 ms)
[ RUN ] SecurityITest.SmokeTestAsAuthorizedUser
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:50:43 dist-test-slave-hxxt krb5kdc[1514](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:50:43 dist-test-slave-hxxt krb5kdc[1514](info): set up 2 sockets
Oct 27 14:50:43 dist-test-slave-hxxt krb5kdc[1514](info): commencing operation
krb5kdc: starting...
W20251027 14:50:45.316726 416 mini_kdc.cc:121] Time spent starting KDC: real 1.814s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:50:45 dist-test-slave-hxxt krb5kdc[1514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:50:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:50:45Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:45.506966 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37547
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37547
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:45.817090 1530 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:45.817387 1530 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:45.817476 1530 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:45.825421 1530 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:50:45.825515 1530 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251027 14:50:45.825563 1530 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:45.825613 1530 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:45.825672 1530 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:50:45.825726 1530 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:50:45.835932 1530 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37547
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37547
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:45.837420 1530 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:45.838624 1530 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:45.846434 1535 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:45.846405 1536 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:45.846907 1538 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:45.847538 1530 server_base.cc:1047] running on GCE node
I20251027 14:50:45.848217 1530 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:45.849529 1530 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:45.850745 1530 hybrid_clock.cc:648] HybridClock initialized: now 1761576645850684 us; error 42 us; skew 500 ppm
I20251027 14:50:45.851106 1530 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:45 dist-test-slave-hxxt krb5kdc[1514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:45.853960 1530 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:50:45.855356 1530 webserver.cc:492] Webserver started at http://127.0.104.62:40077/ using document root <none> and password file <none>
I20251027 14:50:45.855985 1530 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:45.856091 1530 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:45.856367 1530 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:45.858891 1530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "bf16a95f29724fed95c5103324ab0a12"
format_stamp: "Formatted at 2025-10-27 14:50:45 on dist-test-slave-hxxt"
server_key: "597b162472b2be5cf9a0b5e2a09707d9"
server_key_iv: "7b9a086433793b97744340212ddb4029"
server_key_version: "encryptionkey@0"
I20251027 14:50:45.859601 1530 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "bf16a95f29724fed95c5103324ab0a12"
format_stamp: "Formatted at 2025-10-27 14:50:45 on dist-test-slave-hxxt"
server_key: "597b162472b2be5cf9a0b5e2a09707d9"
server_key_iv: "7b9a086433793b97744340212ddb4029"
server_key_version: "encryptionkey@0"
I20251027 14:50:45.864739 1530 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.006s
I20251027 14:50:45.867926 1545 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:45.869192 1530 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:50:45.869336 1530 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "bf16a95f29724fed95c5103324ab0a12"
format_stamp: "Formatted at 2025-10-27 14:50:45 on dist-test-slave-hxxt"
server_key: "597b162472b2be5cf9a0b5e2a09707d9"
server_key_iv: "7b9a086433793b97744340212ddb4029"
server_key_version: "encryptionkey@0"
I20251027 14:50:45.869516 1530 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:45.885439 1530 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:45.892278 1530 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:45.892602 1530 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:45.912758 1530 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37547
I20251027 14:50:45.912791 1606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37547 every 8 connection(s)
I20251027 14:50:45.914156 1530 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:50:45.918089 1607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:45.921442 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1530
I20251027 14:50:45.921663 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:50:45.922382 416 external_mini_cluster.cc:1468] Setting key 73513c0e58989476d38a9fc88abd2df3
I20251027 14:50:45.930035 1607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: Bootstrap starting.
I20251027 14:50:45.934240 1607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: Neither blocks nor log segments found. Creating new log.
Oct 27 14:50:45 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576645, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:45.935272 1607 log.cc:826] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:45.938061 1607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: No bootstrap required, opened a new log
I20251027 14:50:45.942044 1610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:45.924359 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47376 (local address 127.0.104.62:37547)
1027 14:50:45.925093 (+ 734us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:45.925108 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:45.925147 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:50:45.926219 (+ 1072us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:45.926239 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:45.926631 (+ 392us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:45.926994 (+ 363us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:45.928858 (+ 1864us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:45.930168 (+ 1310us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:45.931578 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:45.932073 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:45.935651 (+ 3578us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:45.935729 (+ 78us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:45.935752 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:45.935796 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:45.938604 (+ 2808us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:45.939302 (+ 698us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:45.939309 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:45.939315 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:45.939403 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:45.939694 (+ 291us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:45.939698 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:45.939700 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:45.940129 (+ 429us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:45.940354 (+ 225us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:45.940760 (+ 406us) server_negotiation.cc:300] Negotiation successful
1027 14:50:45.941103 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":335,"threads_started":1}
I20251027 14:50:45.943678 1607 raft_consensus.cc:359] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf16a95f29724fed95c5103324ab0a12" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37547 } }
I20251027 14:50:45.944010 1607 raft_consensus.cc:385] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:45.944113 1607 raft_consensus.cc:740] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf16a95f29724fed95c5103324ab0a12, State: Initialized, Role: FOLLOWER
I20251027 14:50:45.944690 1607 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf16a95f29724fed95c5103324ab0a12" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37547 } }
I20251027 14:50:45.944869 1607 raft_consensus.cc:399] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:50:45.944983 1607 raft_consensus.cc:493] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:50:45.945124 1607 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:45.947269 1607 raft_consensus.cc:515] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf16a95f29724fed95c5103324ab0a12" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37547 } }
I20251027 14:50:45.947715 1607 leader_election.cc:304] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: bf16a95f29724fed95c5103324ab0a12; no voters:
I20251027 14:50:45.948262 1607 leader_election.cc:290] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:50:45.948364 1612 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:45.948977 1612 raft_consensus.cc:697] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [term 1 LEADER]: Becoming Leader. State: Replica: bf16a95f29724fed95c5103324ab0a12, State: Running, Role: LEADER
I20251027 14:50:45.949421 1612 consensus_queue.cc:237] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf16a95f29724fed95c5103324ab0a12" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37547 } }
I20251027 14:50:45.950109 1607 sys_catalog.cc:565] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:50:45.952266 1614 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader bf16a95f29724fed95c5103324ab0a12. Latest consensus state: current_term: 1 leader_uuid: "bf16a95f29724fed95c5103324ab0a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf16a95f29724fed95c5103324ab0a12" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37547 } } }
I20251027 14:50:45.952566 1614 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:45.953006 1613 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "bf16a95f29724fed95c5103324ab0a12" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "bf16a95f29724fed95c5103324ab0a12" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37547 } } }
I20251027 14:50:45.953325 1613 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12 [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:45.954157 1619 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:50:45.959990 1619 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:50:45.968215 1619 catalog_manager.cc:1357] Generated new cluster ID: ee5ef5221cd5418daa8cf5ca2066f5b7
I20251027 14:50:45.968322 1619 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:50:45.979331 1619 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:50:45.980727 1619 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:50:45.989607 1619 catalog_manager.cc:6022] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: Generated new TSK 0
I20251027 14:50:45.990574 1619 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:46.073869 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37547
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:46.391295 1635 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:46.391605 1635 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:46.391753 1635 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:46.401028 1635 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:46.401156 1635 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:46.401379 1635 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:50:46.401530 1635 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:50:46.412420 1635 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:37547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:46.413968 1635 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:46.415261 1635 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:46.424160 1641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:46.424227 1640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:46.424309 1643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:46.425271 1635 server_base.cc:1047] running on GCE node
I20251027 14:50:46.425819 1635 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:46.426681 1635 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:46.427868 1635 hybrid_clock.cc:648] HybridClock initialized: now 1761576646427824 us; error 33 us; skew 500 ppm
I20251027 14:50:46.428143 1635 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:46 dist-test-slave-hxxt krb5kdc[1514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:46.431084 1635 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:50:46.432541 1635 webserver.cc:492] Webserver started at http://127.0.104.1:33755/ using document root <none> and password file <none>
I20251027 14:50:46.433265 1635 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:46.433368 1635 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:46.433706 1635 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:46.436265 1635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "719824530fc143f3975542d473f4d1a8"
format_stamp: "Formatted at 2025-10-27 14:50:46 on dist-test-slave-hxxt"
server_key: "49c0354c9a1f0746019c7aeca39415db"
server_key_iv: "a05325eda12ef3748126c639ef083ccd"
server_key_version: "encryptionkey@0"
I20251027 14:50:46.437063 1635 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "719824530fc143f3975542d473f4d1a8"
format_stamp: "Formatted at 2025-10-27 14:50:46 on dist-test-slave-hxxt"
server_key: "49c0354c9a1f0746019c7aeca39415db"
server_key_iv: "a05325eda12ef3748126c639ef083ccd"
server_key_version: "encryptionkey@0"
I20251027 14:50:46.442355 1635 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:50:46.445803 1650 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:46.447144 1635 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:50:46.447296 1635 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "719824530fc143f3975542d473f4d1a8"
format_stamp: "Formatted at 2025-10-27 14:50:46 on dist-test-slave-hxxt"
server_key: "49c0354c9a1f0746019c7aeca39415db"
server_key_iv: "a05325eda12ef3748126c639ef083ccd"
server_key_version: "encryptionkey@0"
I20251027 14:50:46.447475 1635 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:46.461062 1635 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:46.464777 1635 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:46.465030 1635 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:46.467512 1635 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:46.467617 1635 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:46.467720 1635 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:46.467790 1635 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:50:46 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:46.486521 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:46.468086 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:42419 (local address 127.0.104.62:37547)
1027 14:50:46.468580 (+ 494us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:46.468586 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:46.469648 (+ 1062us) server_negotiation.cc:408] Connection header received
1027 14:50:46.471337 (+ 1689us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:46.471344 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:46.471428 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:46.471563 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:46.473546 (+ 1983us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.474972 (+ 1426us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:46.476591 (+ 1619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.477026 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:46.480278 (+ 3252us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:46.480312 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:46.480317 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:46.480356 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:46.483189 (+ 2833us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:46.484103 (+ 914us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:46.484107 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:46.484109 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:46.484178 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:46.484566 (+ 388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:46.484572 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:46.484576 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:46.484789 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:46.484923 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:46.485883 (+ 960us) server_negotiation.cc:300] Negotiation successful
1027 14:50:46.486143 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":392,"thread_start_us":300,"threads_started":1}
I20251027 14:50:46.487298 1660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:46.468698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.104.1:42419)
1027 14:50:46.469471 (+ 773us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:46.469518 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:46.471056 (+ 1538us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:46.471777 (+ 721us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:46.471785 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:46.472357 (+ 572us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:46.473298 (+ 941us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:46.473324 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.475138 (+ 1814us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:46.475144 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:46.476454 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:46.476466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.476762 (+ 296us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:46.477567 (+ 805us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:46.477594 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:46.480052 (+ 2458us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:46.483361 (+ 3309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:46.483370 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:46.483389 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:46.483929 (+ 540us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:46.484278 (+ 349us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:46.484283 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:46.484287 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:46.484454 (+ 167us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:46.484908 (+ 454us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:46.484919 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:46.485316 (+ 397us) client_negotiation.cc:769] Sending connection context
1027 14:50:46.485783 (+ 467us) client_negotiation.cc:240] Negotiation successful
1027 14:50:46.486194 (+ 411us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":359,"threads_started":1}
I20251027 14:50:46.506430 1714 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:46.492824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.104.1:36959)
1027 14:50:46.493440 (+ 616us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:46.493459 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:46.493614 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:46.493941 (+ 327us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:46.493947 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:46.494595 (+ 648us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:46.495264 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:46.495279 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.496732 (+ 1453us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:46.496737 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:46.497920 (+ 1183us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:46.497936 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.498183 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:46.498962 (+ 779us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:46.498979 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:46.499873 (+ 894us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:46.503536 (+ 3663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:46.503540 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:46.503543 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:46.504001 (+ 458us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:46.504324 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:46.504330 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:46.504334 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:46.504458 (+ 124us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:46.505174 (+ 716us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:46.505179 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:46.505357 (+ 178us) client_negotiation.cc:769] Sending connection context
1027 14:50:46.505527 (+ 170us) client_negotiation.cc:240] Negotiation successful
1027 14:50:46.505890 (+ 363us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":462,"thread_start_us":316,"threads_started":1}
I20251027 14:50:46.507114 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:46.492763 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:36959 (local address 127.0.104.62:37547)
1027 14:50:46.492940 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:46.492946 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:46.493565 (+ 619us) server_negotiation.cc:408] Connection header received
1027 14:50:46.493731 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:46.493736 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:46.493804 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:46.493886 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:46.495457 (+ 1571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.496590 (+ 1133us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:46.498071 (+ 1481us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.498473 (+ 402us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:46.500083 (+ 1610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:46.500122 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:46.500126 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:46.500154 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:46.503407 (+ 3253us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:46.504109 (+ 702us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:46.504115 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:46.504119 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:46.504185 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:46.504620 (+ 435us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:46.504626 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:46.504630 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:46.504957 (+ 327us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:46.505110 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:46.506514 (+ 1404us) server_negotiation.cc:300] Negotiation successful
1027 14:50:46.506790 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251027 14:50:46.517985 1635 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33743
I20251027 14:50:46.518007 1770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33743 every 8 connection(s)
I20251027 14:50:46.519224 1635 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:50:46.521287 1771 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37547
I20251027 14:50:46.521500 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1635
I20251027 14:50:46.521588 1771 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:46.521723 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:50:46.522439 416 external_mini_cluster.cc:1468] Setting key 63ea1f66b0352d6c2bb650c689be3ff1
I20251027 14:50:46.522552 1771 heartbeater.cc:507] Master 127.0.104.62:37547 requested a full tablet report, sending...
I20251027 14:50:46.524737 1561 ts_manager.cc:194] Registered new tserver with Master: 719824530fc143f3975542d473f4d1a8 (127.0.104.1:33743)
I20251027 14:50:46.526237 1561 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:42419
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:46.587083 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37547
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:46.896762 1777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:46.897063 1777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:46.897188 1777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:46.905211 1777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:46.905313 1777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:46.905447 1777 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:50:46.905541 1777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:50:46.915757 1777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:37547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:46.917326 1777 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:46.918581 1777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:46.927472 1782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:46.927829 1783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:46.928732 1785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:46.929440 1777 server_base.cc:1047] running on GCE node
I20251027 14:50:46.930114 1777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:46.930968 1777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:46.932166 1777 hybrid_clock.cc:648] HybridClock initialized: now 1761576646932121 us; error 30 us; skew 500 ppm
I20251027 14:50:46.932483 1777 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:46 dist-test-slave-hxxt krb5kdc[1514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:46.937987 1777 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:50:46.939553 1777 webserver.cc:492] Webserver started at http://127.0.104.2:33503/ using document root <none> and password file <none>
I20251027 14:50:46.940310 1777 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:46.940443 1777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:46.940797 1777 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:46.944018 1777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "8e30348760a041b0b51ffed790ca387e"
format_stamp: "Formatted at 2025-10-27 14:50:46 on dist-test-slave-hxxt"
server_key: "73109e7705bf5eaec6b64d339d4e596f"
server_key_iv: "a2dffefb8b044639b69a3cffbdf2779e"
server_key_version: "encryptionkey@0"
I20251027 14:50:46.945235 1777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "8e30348760a041b0b51ffed790ca387e"
format_stamp: "Formatted at 2025-10-27 14:50:46 on dist-test-slave-hxxt"
server_key: "73109e7705bf5eaec6b64d339d4e596f"
server_key_iv: "a2dffefb8b044639b69a3cffbdf2779e"
server_key_version: "encryptionkey@0"
I20251027 14:50:46.953331 1777 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20251027 14:50:46.958465 1792 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:46.960000 1777 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251027 14:50:46.960160 1777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "8e30348760a041b0b51ffed790ca387e"
format_stamp: "Formatted at 2025-10-27 14:50:46 on dist-test-slave-hxxt"
server_key: "73109e7705bf5eaec6b64d339d4e596f"
server_key_iv: "a2dffefb8b044639b69a3cffbdf2779e"
server_key_version: "encryptionkey@0"
I20251027 14:50:46.960341 1777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:46.978474 1777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:46.983844 1777 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:46.984164 1777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:46.987740 1777 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:46.987856 1777 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:46.987970 1777 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:46.988068 1777 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:50:46 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576646, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:47.009603 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:46.989605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:39191 (local address 127.0.104.62:37547)
1027 14:50:46.990094 (+ 489us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:46.990101 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:46.990126 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:50:46.991830 (+ 1704us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:46.991839 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:46.991929 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:46.992051 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:46.994225 (+ 2174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.995535 (+ 1310us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:46.997626 (+ 2091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.998136 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.002793 (+ 4657us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:47.002833 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:47.002839 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:47.002881 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:47.006062 (+ 3181us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.006942 (+ 880us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.006950 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.006954 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.007049 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.007426 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.007432 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.007435 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.007760 (+ 325us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:47.007888 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.009018 (+ 1130us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.009261 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376}
I20251027 14:50:47.011593 1801 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:46.989008 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.104.2:39191)
1027 14:50:46.989908 (+ 900us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:46.989955 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:46.991520 (+ 1565us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:46.992256 (+ 736us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:46.992270 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:46.992915 (+ 645us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:46.994009 (+ 1094us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:46.994040 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.995755 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:46.995760 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:46.997457 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:46.997482 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:46.997858 (+ 376us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:46.998787 (+ 929us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:46.998837 (+ 50us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:47.002513 (+ 3676us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:47.006226 (+ 3713us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.006237 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.006258 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.006786 (+ 528us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.007167 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.007172 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.007176 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.007301 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.007886 (+ 585us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:47.007898 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:47.008369 (+ 471us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.008907 (+ 538us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.009450 (+ 543us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":1020,"threads_started":1}
I20251027 14:50:47.029891 1777 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:44611
I20251027 14:50:47.029911 1911 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:44611 every 8 connection(s)
I20251027 14:50:47.030903 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.017983 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:52973 (local address 127.0.104.62:37547)
1027 14:50:47.018154 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.018159 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.019017 (+ 858us) server_negotiation.cc:408] Connection header received
1027 14:50:47.019186 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.019191 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.019256 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.019347 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:47.020962 (+ 1615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.021828 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.023376 (+ 1548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.023816 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.025392 (+ 1576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:47.025424 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:47.025427 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:47.025462 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:47.028536 (+ 3074us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.029220 (+ 684us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.029226 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.029230 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.029317 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.029624 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.029629 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.029632 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.029886 (+ 254us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:47.030029 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.030361 (+ 332us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.030580 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:50:47.030989 1881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.018058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.104.2:52973)
1027 14:50:47.018894 (+ 836us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.018916 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.019078 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.019403 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.019409 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.020042 (+ 633us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:47.020807 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.020818 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.022008 (+ 1190us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.022012 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.023236 (+ 1224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.023248 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.023428 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.024248 (+ 820us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:47.024266 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:47.025229 (+ 963us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:47.028674 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.028680 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.028685 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.029039 (+ 354us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.029420 (+ 381us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.029426 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.029428 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.029528 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.029996 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:47.030001 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:47.030139 (+ 138us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.030265 (+ 126us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.030557 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":660,"thread_start_us":437,"threads_started":1}
I20251027 14:50:47.031431 1777 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:50:47.031706 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1777
I20251027 14:50:47.031917 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:50:47.032370 416 external_mini_cluster.cc:1468] Setting key 593ab45d2f957484ec9c6719b7647345
I20251027 14:50:47.034262 1912 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37547
I20251027 14:50:47.034986 1912 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:47.035800 1912 heartbeater.cc:507] Master 127.0.104.62:37547 requested a full tablet report, sending...
I20251027 14:50:47.037212 1561 ts_manager.cc:194] Registered new tserver with Master: 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611)
I20251027 14:50:47.037986 1561 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:39191
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:47.099988 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37547
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46457
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:47.410822 1918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:47.411119 1918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:47.411252 1918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:47.419305 1918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:47.419413 1918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:47.419556 1918 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:50:47.419665 1918 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:50:47.430002 1918 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46457
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:37547
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:47.431528 1918 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:47.432868 1918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:47.442159 1923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:47.442193 1924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:47.442976 1918 server_base.cc:1047] running on GCE node
W20251027 14:50:47.443490 1926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:47.444109 1918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:47.445060 1918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:47.446246 1918 hybrid_clock.cc:648] HybridClock initialized: now 1761576647446192 us; error 48 us; skew 500 ppm
I20251027 14:50:47.446524 1918 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:47 dist-test-slave-hxxt krb5kdc[1514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:47.449771 1918 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:50:47.451313 1918 webserver.cc:492] Webserver started at http://127.0.104.3:35251/ using document root <none> and password file <none>
I20251027 14:50:47.452066 1918 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:47.452163 1918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:47.452529 1918 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:47.455082 1918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "2fb3bb11c72844d98bb162991bda140f"
format_stamp: "Formatted at 2025-10-27 14:50:47 on dist-test-slave-hxxt"
server_key: "43a1f49171eda40756887d73a032bf0b"
server_key_iv: "e37dc4614e2bf97625e44ee62c62bbc4"
server_key_version: "encryptionkey@0"
I20251027 14:50:47.455809 1918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "2fb3bb11c72844d98bb162991bda140f"
format_stamp: "Formatted at 2025-10-27 14:50:47 on dist-test-slave-hxxt"
server_key: "43a1f49171eda40756887d73a032bf0b"
server_key_iv: "e37dc4614e2bf97625e44ee62c62bbc4"
server_key_version: "encryptionkey@0"
I20251027 14:50:47.461076 1918 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251027 14:50:47.464215 1933 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:47.465469 1918 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:50:47.465613 1918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "2fb3bb11c72844d98bb162991bda140f"
format_stamp: "Formatted at 2025-10-27 14:50:47 on dist-test-slave-hxxt"
server_key: "43a1f49171eda40756887d73a032bf0b"
server_key_iv: "e37dc4614e2bf97625e44ee62c62bbc4"
server_key_version: "encryptionkey@0"
I20251027 14:50:47.465788 1918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:47.481518 1918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:47.484972 1918 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:47.485227 1918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:47.487856 1918 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:47.487954 1918 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:47.488068 1918 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:47.488142 1918 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:50:47 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:47.509634 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.488582 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:49285 (local address 127.0.104.62:37547)
1027 14:50:47.488794 (+ 212us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.488801 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.490178 (+ 1377us) server_negotiation.cc:408] Connection header received
1027 14:50:47.492401 (+ 2223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.492409 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.492482 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.492566 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:47.495219 (+ 2653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.496502 (+ 1283us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.498079 (+ 1577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.498536 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.502515 (+ 3979us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:47.502554 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:47.502557 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:47.502589 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:47.505825 (+ 3236us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.506762 (+ 937us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.506768 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.506772 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.506855 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.507316 (+ 461us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.507320 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.507322 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.507601 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:47.507775 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.509034 (+ 1259us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.509293 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":85}
I20251027 14:50:47.511267 1942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.489137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.104.3:49285)
1027 14:50:47.489979 (+ 842us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.490034 (+ 55us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.492059 (+ 2025us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.492871 (+ 812us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.492893 (+ 22us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.493743 (+ 850us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:47.495004 (+ 1261us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.495035 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.496679 (+ 1644us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.496684 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.497938 (+ 1254us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.497954 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.498235 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.499080 (+ 845us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:47.499115 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:47.502297 (+ 3182us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:47.505971 (+ 3674us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.505980 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.506001 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.506630 (+ 629us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.506972 (+ 342us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.506978 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.506981 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.507161 (+ 180us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.507776 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:47.507786 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:47.508343 (+ 557us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.508913 (+ 570us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.509502 (+ 589us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":542,"thread_start_us":365,"threads_started":1}
I20251027 14:50:47.529271 1771 heartbeater.cc:499] Master 127.0.104.62:37547 was elected leader, sending a full tablet report...
I20251027 14:50:47.529556 1918 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46203
I20251027 14:50:47.529605 2052 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46203 every 8 connection(s)
I20251027 14:50:47.531268 1918 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:50:47.532274 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.518332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:51541 (local address 127.0.104.62:37547)
1027 14:50:47.518510 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.518516 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.519107 (+ 591us) server_negotiation.cc:408] Connection header received
1027 14:50:47.519257 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.519264 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.519356 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.519464 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:47.521050 (+ 1586us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.522407 (+ 1357us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.523982 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.524504 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.526039 (+ 1535us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:47.526091 (+ 52us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:47.526097 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:47.526136 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:47.529432 (+ 3296us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.530233 (+ 801us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.530238 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.530241 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.530322 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.530809 (+ 487us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.530814 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.530817 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.531159 (+ 342us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:47.531304 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.531702 (+ 398us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.531958 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:50:47.532330 2035 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.518346 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.104.3:51541)
1027 14:50:47.518978 (+ 632us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.518998 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.519139 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.519503 (+ 364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.519509 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.520113 (+ 604us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:47.520832 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.520848 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.522633 (+ 1785us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.522639 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.523821 (+ 1182us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.523837 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.524058 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.524846 (+ 788us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:47.524875 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:47.525871 (+ 996us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:47.529600 (+ 3729us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.529606 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.529609 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.530061 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.530444 (+ 383us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:47.530449 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:47.530452 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:47.530552 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:47.531299 (+ 747us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:47.531305 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:47.531467 (+ 162us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.531584 (+ 117us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.531858 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":355,"threads_started":1}
I20251027 14:50:47.534211 2053 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37547
I20251027 14:50:47.534490 2053 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:47.535415 2053 heartbeater.cc:507] Master 127.0.104.62:37547 requested a full tablet report, sending...
I20251027 14:50:47.536935 1562 ts_manager.cc:194] Registered new tserver with Master: 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203)
I20251027 14:50:47.537832 1562 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:49285
I20251027 14:50:47.538575 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 1918
I20251027 14:50:47.538764 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:50:47.539122 416 external_mini_cluster.cc:1468] Setting key 698bdebb5bc78e2d7ca257598a189521
I20251027 14:50:47.541872 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 27 14:50:47 dist-test-slave-hxxt krb5kdc[1514](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 27 14:50:47 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:47.583981 1659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.571385 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47392 (local address 127.0.104.62:37547)
1027 14:50:47.571532 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.571537 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.571860 (+ 323us) server_negotiation.cc:408] Connection header received
1027 14:50:47.572019 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.572024 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.572090 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.572180 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:47.573682 (+ 1502us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.574698 (+ 1016us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.575790 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.576176 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.579235 (+ 3059us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:47.579264 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:47.579268 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:47.579301 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:47.581721 (+ 2420us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.582328 (+ 607us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.582333 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.582336 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.582408 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:47.582681 (+ 273us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:47.582687 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:47.582690 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:47.582946 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:47.583171 (+ 225us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.583435 (+ 264us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.583663 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:50:47.587699 1562 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:50:47.592274 1562 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:47.626976 2069 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.605444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33743 (local address 127.0.0.1:36612)
1027 14:50:47.619290 (+ 13846us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.619310 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.619446 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.620558 (+ 1112us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.620567 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.620608 (+ 41us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:47.621512 (+ 904us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.621527 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.624459 (+ 2932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.624465 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.625959 (+ 1494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.625970 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.626137 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.626214 (+ 77us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.626381 (+ 167us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.626578 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13736,"thread_start_us":15518,"threads_started":1}
I20251027 14:50:47.627009 2066 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.604358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:46203 (local address 127.0.0.1:40276)
1027 14:50:47.605515 (+ 1157us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.605551 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.605736 (+ 185us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.622133 (+ 16397us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.622138 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.622163 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:47.622786 (+ 623us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.622796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.624628 (+ 1832us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.624634 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.626299 (+ 1665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.626312 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.626492 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.626520 (+ 28us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.626593 (+ 73us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.626701 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":888,"thread_start_us":785,"threads_started":1}
I20251027 14:50:47.627571 2068 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.606036 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:44611 (local address 127.0.0.1:35200)
1027 14:50:47.621290 (+ 15254us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.621313 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.621528 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.622413 (+ 885us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.622419 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.622448 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:47.623178 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.623190 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.625206 (+ 2016us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.625215 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.626912 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.626925 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.627094 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.627116 (+ 22us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.627180 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.627279 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15118,"thread_start_us":13037,"threads_started":1}
I20251027 14:50:47.628036 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.606091 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36612 (local address 127.0.104.1:33743)
1027 14:50:47.619749 (+ 13658us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.619761 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.619797 (+ 36us) server_negotiation.cc:408] Connection header received
1027 14:50:47.619922 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.619933 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.620257 (+ 324us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.620544 (+ 287us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:47.621720 (+ 1176us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.623662 (+ 1942us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.626112 (+ 2450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.627162 (+ 1050us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.627488 (+ 326us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.627615 (+ 127us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.627795 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13445,"thread_start_us":13340,"threads_started":1}
I20251027 14:50:47.628968 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.604741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40276 (local address 127.0.104.3:46203)
1027 14:50:47.621673 (+ 16932us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.621681 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.621705 (+ 24us) server_negotiation.cc:408] Connection header received
1027 14:50:47.621777 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.621782 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.621959 (+ 177us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.622137 (+ 178us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:47.622938 (+ 801us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.624431 (+ 1493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.627142 (+ 2711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.628258 (+ 1116us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.628429 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.628546 (+ 117us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.628708 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16741,"thread_start_us":16563,"threads_started":1}
I20251027 14:50:47.630143 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.606294 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35200 (local address 127.0.104.2:44611)
1027 14:50:47.618654 (+ 12360us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.618666 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.621433 (+ 2767us) server_negotiation.cc:408] Connection header received
1027 14:50:47.621714 (+ 281us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.621723 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.622005 (+ 282us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.622201 (+ 196us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:47.623383 (+ 1182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.624969 (+ 1586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.627767 (+ 2798us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.628863 (+ 1096us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.628994 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.629718 (+ 724us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.629905 (+ 187us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12174,"thread_start_us":12383,"threads_started":1}
I20251027 14:50:47.632647 1983 tablet_service.cc:1505] Processing CreateTablet for tablet bce1e475afce4afd9c20b57f08209180 (DEFAULT_TABLE table=test-table [id=0dcb0070991d457da7c2a9d9292b93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:47.632556 1701 tablet_service.cc:1505] Processing CreateTablet for tablet bce1e475afce4afd9c20b57f08209180 (DEFAULT_TABLE table=test-table [id=0dcb0070991d457da7c2a9d9292b93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:47.633847 1983 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce1e475afce4afd9c20b57f08209180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:47.633906 1701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce1e475afce4afd9c20b57f08209180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:47.634044 1842 tablet_service.cc:1505] Processing CreateTablet for tablet bce1e475afce4afd9c20b57f08209180 (DEFAULT_TABLE table=test-table [id=0dcb0070991d457da7c2a9d9292b93ac]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:47.635378 1842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bce1e475afce4afd9c20b57f08209180. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:47.644400 2072 tablet_bootstrap.cc:492] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Bootstrap starting.
I20251027 14:50:47.647392 2073 tablet_bootstrap.cc:492] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Bootstrap starting.
I20251027 14:50:47.648802 2072 tablet_bootstrap.cc:654] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:47.648830 2074 tablet_bootstrap.cc:492] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Bootstrap starting.
I20251027 14:50:47.650658 2072 log.cc:826] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:47.651537 2073 tablet_bootstrap.cc:654] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:47.651690 2074 tablet_bootstrap.cc:654] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:47.652777 2073 log.cc:826] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:47.652876 2074 log.cc:826] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:47.654242 2072 tablet_bootstrap.cc:492] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: No bootstrap required, opened a new log
I20251027 14:50:47.654551 2072 ts_tablet_manager.cc:1403] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Time spent bootstrapping tablet: real 0.010s user 0.006s sys 0.000s
I20251027 14:50:47.655987 2073 tablet_bootstrap.cc:492] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: No bootstrap required, opened a new log
I20251027 14:50:47.656260 2073 ts_tablet_manager.cc:1403] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20251027 14:50:47.656450 2074 tablet_bootstrap.cc:492] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: No bootstrap required, opened a new log
I20251027 14:50:47.656697 2074 ts_tablet_manager.cc:1403] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Time spent bootstrapping tablet: real 0.008s user 0.006s sys 0.002s
I20251027 14:50:47.659875 2072 raft_consensus.cc:359] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.660220 2072 raft_consensus.cc:385] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:47.660332 2072 raft_consensus.cc:740] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb3bb11c72844d98bb162991bda140f, State: Initialized, Role: FOLLOWER
I20251027 14:50:47.660970 2072 consensus_queue.cc:260] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.661989 2053 heartbeater.cc:499] Master 127.0.104.62:37547 was elected leader, sending a full tablet report...
I20251027 14:50:47.662595 2072 ts_tablet_manager.cc:1434] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Time spent starting tablet: real 0.008s user 0.006s sys 0.003s
I20251027 14:50:47.663348 2073 raft_consensus.cc:359] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.663779 2073 raft_consensus.cc:385] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:47.663892 2073 raft_consensus.cc:740] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719824530fc143f3975542d473f4d1a8, State: Initialized, Role: FOLLOWER
I20251027 14:50:47.663913 2074 raft_consensus.cc:359] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.664287 2074 raft_consensus.cc:385] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:47.664402 2074 raft_consensus.cc:740] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e30348760a041b0b51ffed790ca387e, State: Initialized, Role: FOLLOWER
I20251027 14:50:47.664557 2073 consensus_queue.cc:260] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.665009 2074 consensus_queue.cc:260] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.666270 2073 ts_tablet_manager.cc:1434] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20251027 14:50:47.667186 1912 heartbeater.cc:499] Master 127.0.104.62:37547 was elected leader, sending a full tablet report...
I20251027 14:50:47.667706 2074 ts_tablet_manager.cc:1434] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Time spent starting tablet: real 0.011s user 0.008s sys 0.002s
W20251027 14:50:47.771347 1772 tablet.cc:2378] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:50:47.784029 2054 tablet.cc:2378] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:50:47.784571 1913 tablet.cc:2378] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:50:47.908636 2080 raft_consensus.cc:493] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:47.908911 2080 raft_consensus.cc:515] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.910602 2080 leader_election.cc:290] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203), 719824530fc143f3975542d473f4d1a8 (127.0.104.1:33743)
I20251027 14:50:47.917572 2081 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.910910 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:46203 (local address 127.0.104.2:50585)
1027 14:50:47.912230 (+ 1320us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.912252 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.912430 (+ 178us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.912939 (+ 509us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.912943 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.912971 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:47.913718 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.913728 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.915632 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.915637 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.916826 (+ 1189us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.916838 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.917011 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.917039 (+ 28us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.917108 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.917229 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1186,"thread_start_us":996,"threads_started":1}
I20251027 14:50:47.918301 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.911003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:50585 (local address 127.0.104.3:46203)
1027 14:50:47.911174 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.911180 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.912351 (+ 1171us) server_negotiation.cc:408] Connection header received
1027 14:50:47.912589 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.912595 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.912684 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.912746 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:47.913867 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.915364 (+ 1497us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.916977 (+ 1613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.917966 (+ 989us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.918007 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.918072 (+ 65us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.918154 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:50:47.919181 2082 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.910966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33743 (local address 127.0.104.2:50513)
1027 14:50:47.912312 (+ 1346us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:47.912329 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:47.912549 (+ 220us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:47.912885 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:47.912890 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:47.912915 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:47.913785 (+ 870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.913796 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.915563 (+ 1767us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.915569 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:47.917289 (+ 1720us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:47.917302 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.918698 (+ 1396us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.918720 (+ 22us) client_negotiation.cc:769] Sending connection context
1027 14:50:47.918795 (+ 75us) client_negotiation.cc:240] Negotiation successful
1027 14:50:47.918907 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1243,"spinlock_wait_cycles":896,"thread_start_us":1082,"threads_started":1}
I20251027 14:50:47.919274 2003 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bce1e475afce4afd9c20b57f08209180" candidate_uuid: "8e30348760a041b0b51ffed790ca387e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3bb11c72844d98bb162991bda140f" is_pre_election: true
I20251027 14:50:47.919510 2070 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.911073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:50513 (local address 127.0.104.1:33743)
1027 14:50:47.911220 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.911225 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.912509 (+ 1284us) server_negotiation.cc:408] Connection header received
1027 14:50:47.912686 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.912692 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.912762 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.912844 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:47.913935 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.915364 (+ 1429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.917460 (+ 2096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.919065 (+ 1605us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.919114 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.919190 (+ 76us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.919305 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:50:47.919760 2003 raft_consensus.cc:2468] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e30348760a041b0b51ffed790ca387e in term 0.
I20251027 14:50:47.920459 1795 leader_election.cc:304] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb3bb11c72844d98bb162991bda140f, 8e30348760a041b0b51ffed790ca387e; no voters:
I20251027 14:50:47.920589 1726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bce1e475afce4afd9c20b57f08209180" candidate_uuid: "8e30348760a041b0b51ffed790ca387e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719824530fc143f3975542d473f4d1a8" is_pre_election: true
I20251027 14:50:47.920820 2080 raft_consensus.cc:2804] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:47.920925 2080 raft_consensus.cc:493] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:47.920986 2080 raft_consensus.cc:3060] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:47.921016 1726 raft_consensus.cc:2468] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8e30348760a041b0b51ffed790ca387e in term 0.
I20251027 14:50:47.923267 2080 raft_consensus.cc:515] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.923877 2080 leader_election.cc:290] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [CANDIDATE]: Term 1 election: Requested vote from peers 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203), 719824530fc143f3975542d473f4d1a8 (127.0.104.1:33743)
I20251027 14:50:47.924197 2003 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bce1e475afce4afd9c20b57f08209180" candidate_uuid: "8e30348760a041b0b51ffed790ca387e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3bb11c72844d98bb162991bda140f"
I20251027 14:50:47.924333 1726 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bce1e475afce4afd9c20b57f08209180" candidate_uuid: "8e30348760a041b0b51ffed790ca387e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "719824530fc143f3975542d473f4d1a8"
I20251027 14:50:47.924402 2003 raft_consensus.cc:3060] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:47.924540 1726 raft_consensus.cc:3060] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:47.926729 1726 raft_consensus.cc:2468] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e30348760a041b0b51ffed790ca387e in term 1.
I20251027 14:50:47.927209 1796 leader_election.cc:304] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 719824530fc143f3975542d473f4d1a8, 8e30348760a041b0b51ffed790ca387e; no voters:
I20251027 14:50:47.927476 2080 raft_consensus.cc:2804] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:47.928305 2003 raft_consensus.cc:2468] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8e30348760a041b0b51ffed790ca387e in term 1.
I20251027 14:50:47.928690 2080 raft_consensus.cc:697] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 1 LEADER]: Becoming Leader. State: Replica: 8e30348760a041b0b51ffed790ca387e, State: Running, Role: LEADER
I20251027 14:50:47.929231 2080 consensus_queue.cc:237] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } }
I20251027 14:50:47.933763 1561 catalog_manager.cc:5649] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e reported cstate change: term changed from 0 to 1, leader changed from <none> to 8e30348760a041b0b51ffed790ca387e (127.0.104.2). New cstate: current_term: 1 leader_uuid: "8e30348760a041b0b51ffed790ca387e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } health_report { overall_health: UNKNOWN } } }
I20251027 14:50:47.987986 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:47.982553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35202 (local address 127.0.104.2:44611)
1027 14:50:47.982733 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:47.982739 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:47.982859 (+ 120us) server_negotiation.cc:408] Connection header received
1027 14:50:47.983169 (+ 310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:47.983174 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:47.983256 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:47.983381 (+ 125us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:47.983906 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.984802 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:47.986371 (+ 1569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:47.986770 (+ 399us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:47.986848 (+ 78us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:47.987376 (+ 528us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:47.987510 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:47.987649 (+ 139us) server_negotiation.cc:300] Negotiation successful
1027 14:50:47.987747 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:50:47.995450 2003 raft_consensus.cc:1275] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Refusing update from remote peer 8e30348760a041b0b51ffed790ca387e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:47.995450 1726 raft_consensus.cc:1275] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Refusing update from remote peer 8e30348760a041b0b51ffed790ca387e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:47.996397 2083 consensus_queue.cc:1048] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [LEADER]: Connected to new peer: Peer: permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:47.996665 2080 consensus_queue.cc:1048] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:48.007730 2086 mvcc.cc:204] Tried to move back new op lower bound from 7215417950180429824 to 7215417949922316288. Current Snapshot: MvccSnapshot[applied={T|T < 7215417950180429824}]
I20251027 14:50:48.008507 2088 mvcc.cc:204] Tried to move back new op lower bound from 7215417950180429824 to 7215417949922316288. Current Snapshot: MvccSnapshot[applied={T|T < 7215417950180429824}]
I20251027 14:50:48.024976 1562 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47392:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:48.025174 1562 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47392:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:48.028477 1562 catalog_manager.cc:5936] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: Sending DeleteTablet for 3 replicas of tablet bce1e475afce4afd9c20b57f08209180
I20251027 14:50:48.029683 1983 tablet_service.cc:1552] Processing DeleteTablet for tablet bce1e475afce4afd9c20b57f08209180 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40276
I20251027 14:50:48.029935 1842 tablet_service.cc:1552] Processing DeleteTablet for tablet bce1e475afce4afd9c20b57f08209180 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:35200
I20251027 14:50:48.030268 1701 tablet_service.cc:1552] Processing DeleteTablet for tablet bce1e475afce4afd9c20b57f08209180 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:36612
I20251027 14:50:48.034291 1562 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47392:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:50:48.036237 1562 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:48.043977 2098 tablet_replica.cc:333] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: stopping tablet replica
I20251027 14:50:48.045604 2098 raft_consensus.cc:2243] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:48.046705 2097 tablet_replica.cc:333] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: stopping tablet replica
I20251027 14:50:48.047268 2098 raft_consensus.cc:2272] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:48.047430 1982 tablet_service.cc:1505] Processing CreateTablet for tablet 166ec1eb718344e3b7a9b4fa3ce38a13 (DEFAULT_TABLE table=test-table [id=b16a569f40764cb9b236f0b9768ebbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:48.047881 1982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166ec1eb718344e3b7a9b4fa3ce38a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:48.048511 2097 raft_consensus.cc:2243] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:50:48.049157 2097 raft_consensus.cc:2272] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:48.053833 1842 tablet_service.cc:1505] Processing CreateTablet for tablet 166ec1eb718344e3b7a9b4fa3ce38a13 (DEFAULT_TABLE table=test-table [id=b16a569f40764cb9b236f0b9768ebbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:48.054041 2097 ts_tablet_manager.cc:1916] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:48.056433 1842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166ec1eb718344e3b7a9b4fa3ce38a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:48.056681 2098 ts_tablet_manager.cc:1916] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:48.062711 2096 tablet_replica.cc:333] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: stopping tablet replica
I20251027 14:50:48.063141 2096 raft_consensus.cc:2243] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:48.063591 2096 raft_consensus.cc:2272] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:48.064672 2098 ts_tablet_manager.cc:1929] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:50:48.064867 2098 log.cc:1199] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/bce1e475afce4afd9c20b57f08209180
I20251027 14:50:48.065337 2098 ts_tablet_manager.cc:1950] T bce1e475afce4afd9c20b57f08209180 P 719824530fc143f3975542d473f4d1a8: Deleting consensus metadata
I20251027 14:50:48.067337 1549 catalog_manager.cc:4985] TS 719824530fc143f3975542d473f4d1a8 (127.0.104.1:33743): tablet bce1e475afce4afd9c20b57f08209180 (table test-table [id=0dcb0070991d457da7c2a9d9292b93ac]) successfully deleted
I20251027 14:50:48.067966 2072 tablet_bootstrap.cc:492] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Bootstrap starting.
I20251027 14:50:48.068778 2097 ts_tablet_manager.cc:1929] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:50:48.069044 2097 log.cc:1199] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/bce1e475afce4afd9c20b57f08209180
I20251027 14:50:48.069594 2097 ts_tablet_manager.cc:1950] T bce1e475afce4afd9c20b57f08209180 P 8e30348760a041b0b51ffed790ca387e: Deleting consensus metadata
I20251027 14:50:48.069759 2072 tablet_bootstrap.cc:654] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:48.069737 1701 tablet_service.cc:1505] Processing CreateTablet for tablet 166ec1eb718344e3b7a9b4fa3ce38a13 (DEFAULT_TABLE table=test-table [id=b16a569f40764cb9b236f0b9768ebbe1]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:48.070147 1701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 166ec1eb718344e3b7a9b4fa3ce38a13. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:48.071353 1548 catalog_manager.cc:4985] TS 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611): tablet bce1e475afce4afd9c20b57f08209180 (table test-table [id=0dcb0070991d457da7c2a9d9292b93ac]) successfully deleted
I20251027 14:50:48.073022 2074 tablet_bootstrap.cc:492] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Bootstrap starting.
I20251027 14:50:48.074801 2074 tablet_bootstrap.cc:654] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:48.076699 2073 tablet_bootstrap.cc:492] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Bootstrap starting.
I20251027 14:50:48.078258 2096 ts_tablet_manager.cc:1916] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:48.078339 2073 tablet_bootstrap.cc:654] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:48.080556 2073 tablet_bootstrap.cc:492] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: No bootstrap required, opened a new log
I20251027 14:50:48.080756 2073 ts_tablet_manager.cc:1403] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Time spent bootstrapping tablet: real 0.004s user 0.002s sys 0.001s
I20251027 14:50:48.080922 2074 tablet_bootstrap.cc:492] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: No bootstrap required, opened a new log
I20251027 14:50:48.081166 2074 ts_tablet_manager.cc:1403] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Time spent bootstrapping tablet: real 0.008s user 0.004s sys 0.001s
I20251027 14:50:48.081588 2073 raft_consensus.cc:359] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.081869 2073 raft_consensus.cc:385] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:48.082011 2073 raft_consensus.cc:740] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719824530fc143f3975542d473f4d1a8, State: Initialized, Role: FOLLOWER
I20251027 14:50:48.082060 2074 raft_consensus.cc:359] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.082335 2074 raft_consensus.cc:385] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:48.082473 2074 raft_consensus.cc:740] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e30348760a041b0b51ffed790ca387e, State: Initialized, Role: FOLLOWER
I20251027 14:50:48.082350 2073 consensus_queue.cc:260] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.083864 2074 consensus_queue.cc:260] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.084466 2073 ts_tablet_manager.cc:1434] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20251027 14:50:48.084880 2096 ts_tablet_manager.cc:1929] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:50:48.084986 2096 log.cc:1199] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/bce1e475afce4afd9c20b57f08209180
I20251027 14:50:48.085343 2096 ts_tablet_manager.cc:1950] T bce1e475afce4afd9c20b57f08209180 P 2fb3bb11c72844d98bb162991bda140f: Deleting consensus metadata
I20251027 14:50:48.086531 2074 ts_tablet_manager.cc:1434] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 14:50:48.087127 1548 catalog_manager.cc:4985] TS 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203): tablet bce1e475afce4afd9c20b57f08209180 (table test-table [id=0dcb0070991d457da7c2a9d9292b93ac]) successfully deleted
I20251027 14:50:48.089736 2072 tablet_bootstrap.cc:492] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: No bootstrap required, opened a new log
I20251027 14:50:48.089872 2072 ts_tablet_manager.cc:1403] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Time spent bootstrapping tablet: real 0.022s user 0.002s sys 0.006s
I20251027 14:50:48.090659 2072 raft_consensus.cc:359] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.090857 2072 raft_consensus.cc:385] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:48.090955 2072 raft_consensus.cc:740] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb3bb11c72844d98bb162991bda140f, State: Initialized, Role: FOLLOWER
I20251027 14:50:48.091164 2072 consensus_queue.cc:260] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.091576 2072 ts_tablet_manager.cc:1434] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 14:50:48.338913 2079 raft_consensus.cc:493] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:48.339160 2079 raft_consensus.cc:515] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.341284 2079 leader_election.cc:290] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611), 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203)
I20251027 14:50:48.347492 2103 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.342150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:46203 (local address 127.0.104.1:35753)
1027 14:50:48.343084 (+ 934us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:48.343102 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:48.343265 (+ 163us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:48.343612 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:48.343618 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:48.343645 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:48.344288 (+ 643us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.344302 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.345704 (+ 1402us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.345708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:48.346823 (+ 1115us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.346842 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.346979 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.347005 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:50:48.347071 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:50:48.347186 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":816,"thread_start_us":657,"threads_started":1}
I20251027 14:50:48.348719 2102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.341465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:44611 (local address 127.0.104.1:49079)
1027 14:50:48.341993 (+ 528us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:48.342010 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:48.342146 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:48.342735 (+ 589us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:48.342744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:48.342776 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:48.343559 (+ 783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.343572 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.345129 (+ 1557us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.345135 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:48.346857 (+ 1722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.346872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.348176 (+ 1304us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.348202 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:50:48.348288 (+ 86us) client_negotiation.cc:240] Negotiation successful
1027 14:50:48.348428 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":414,"thread_start_us":290,"threads_started":1}
I20251027 14:50:48.348794 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.341498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:49079 (local address 127.0.104.2:44611)
1027 14:50:48.341669 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.341676 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.342115 (+ 439us) server_negotiation.cc:408] Connection header received
1027 14:50:48.342323 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.342330 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.342406 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.342480 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:48.343754 (+ 1274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.344921 (+ 1167us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.347050 (+ 2129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.348121 (+ 1071us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.348164 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.348354 (+ 190us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.348519 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:50:48.349227 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.342449 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:35753 (local address 127.0.104.3:46203)
1027 14:50:48.342618 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.342624 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.343212 (+ 588us) server_negotiation.cc:408] Connection header received
1027 14:50:48.343401 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.343407 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.343478 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.343570 (+ 92us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:48.344475 (+ 905us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.345515 (+ 1040us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.346980 (+ 1465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.348633 (+ 1653us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.348682 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.348763 (+ 81us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.348985 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:50:48.349682 2003 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "166ec1eb718344e3b7a9b4fa3ce38a13" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3bb11c72844d98bb162991bda140f" is_pre_election: true
I20251027 14:50:48.349893 2003 raft_consensus.cc:2468] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 0.
I20251027 14:50:48.349839 1862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "166ec1eb718344e3b7a9b4fa3ce38a13" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e30348760a041b0b51ffed790ca387e" is_pre_election: true
I20251027 14:50:48.350258 1862 raft_consensus.cc:2468] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 0.
I20251027 14:50:48.350358 1653 leader_election.cc:304] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb3bb11c72844d98bb162991bda140f, 719824530fc143f3975542d473f4d1a8; no voters:
I20251027 14:50:48.350709 2079 raft_consensus.cc:2804] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:48.350838 2079 raft_consensus.cc:493] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:48.350935 2079 raft_consensus.cc:3060] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:48.353327 2079 raft_consensus.cc:515] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.353984 2079 leader_election.cc:290] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 election: Requested vote from peers 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611), 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203)
I20251027 14:50:48.354532 2003 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "166ec1eb718344e3b7a9b4fa3ce38a13" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3bb11c72844d98bb162991bda140f"
I20251027 14:50:48.354543 1862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "166ec1eb718344e3b7a9b4fa3ce38a13" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e30348760a041b0b51ffed790ca387e"
I20251027 14:50:48.354727 2003 raft_consensus.cc:3060] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:48.354738 1862 raft_consensus.cc:3060] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:48.357640 2003 raft_consensus.cc:2468] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 1.
I20251027 14:50:48.358095 1653 leader_election.cc:304] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2fb3bb11c72844d98bb162991bda140f, 719824530fc143f3975542d473f4d1a8; no voters:
I20251027 14:50:48.358387 2079 raft_consensus.cc:2804] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:48.358503 1862 raft_consensus.cc:2468] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 1.
I20251027 14:50:48.358855 2079 raft_consensus.cc:697] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 1 LEADER]: Becoming Leader. State: Replica: 719824530fc143f3975542d473f4d1a8, State: Running, Role: LEADER
I20251027 14:50:48.359329 2079 consensus_queue.cc:237] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.363214 1561 catalog_manager.cc:5649] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 719824530fc143f3975542d473f4d1a8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } health_report { overall_health: UNKNOWN } } }
Oct 27 14:50:48 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576645, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:48.480125 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.467240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37547 (local address 127.0.0.1:47402)
1027 14:50:48.468848 (+ 1608us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:48.468868 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:48.469013 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:48.469542 (+ 529us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:48.469547 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:48.470294 (+ 747us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:48.471157 (+ 863us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.471167 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.472425 (+ 1258us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.472430 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:48.473161 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.473172 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.473293 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.473936 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:48.473953 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:48.475831 (+ 1878us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:48.478389 (+ 2558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:48.478398 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:48.478403 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:48.478717 (+ 314us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:48.478975 (+ 258us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:48.478978 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:48.478980 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:48.479065 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:48.479437 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:48.479442 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:48.479567 (+ 125us) client_negotiation.cc:769] Sending connection context
1027 14:50:48.479677 (+ 110us) client_negotiation.cc:240] Negotiation successful
1027 14:50:48.479869 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1437,"spinlock_wait_cycles":12160,"thread_start_us":1306,"threads_started":1}
I20251027 14:50:48.480319 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.467240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47402 (local address 127.0.104.62:37547)
1027 14:50:48.469243 (+ 2003us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.469247 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.469265 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:50:48.469309 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.469313 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.469367 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.469484 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:48.471301 (+ 1817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.472258 (+ 957us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.473284 (+ 1026us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.473627 (+ 343us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.475971 (+ 2344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:48.476014 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:48.476018 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:48.476045 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:48.478271 (+ 2226us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.478810 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.478814 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.478817 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.478890 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.479141 (+ 251us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.479144 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.479146 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.479346 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:48.479443 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.479804 (+ 361us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.480004 (+ 200us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1917,"thread_start_us":1800,"threads_started":1}
I20251027 14:50:48.484781 1561 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:47402:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251027 14:50:48.485674 1561 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:48.491915 1701 tablet_service.cc:1505] Processing CreateTablet for tablet a671deb6e1514cd0821485590303252b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e0eaae8f03e440eafa4b4d7259cfd48]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:50:48.492247 1842 tablet_service.cc:1505] Processing CreateTablet for tablet a671deb6e1514cd0821485590303252b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e0eaae8f03e440eafa4b4d7259cfd48]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:50:48.492497 1701 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a671deb6e1514cd0821485590303252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:48.492708 1842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a671deb6e1514cd0821485590303252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:48.492679 1982 tablet_service.cc:1505] Processing CreateTablet for tablet a671deb6e1514cd0821485590303252b (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=7e0eaae8f03e440eafa4b4d7259cfd48]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:50:48.493136 1982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a671deb6e1514cd0821485590303252b. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:48.498402 2074 tablet_bootstrap.cc:492] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: Bootstrap starting.
I20251027 14:50:48.499625 2074 tablet_bootstrap.cc:654] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:48.500231 2072 tablet_bootstrap.cc:492] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: Bootstrap starting.
I20251027 14:50:48.500238 2073 tablet_bootstrap.cc:492] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: Bootstrap starting.
I20251027 14:50:48.501873 2073 tablet_bootstrap.cc:654] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:48.502200 2074 tablet_bootstrap.cc:492] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: No bootstrap required, opened a new log
I20251027 14:50:48.502310 2072 tablet_bootstrap.cc:654] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:48.502352 2074 ts_tablet_manager.cc:1403] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:50:48.503013 2074 raft_consensus.cc:359] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.503213 2074 raft_consensus.cc:385] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:48.503291 2074 raft_consensus.cc:740] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8e30348760a041b0b51ffed790ca387e, State: Initialized, Role: FOLLOWER
I20251027 14:50:48.503527 2074 consensus_queue.cc:260] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.503849 2073 tablet_bootstrap.cc:492] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: No bootstrap required, opened a new log
I20251027 14:50:48.503998 2073 ts_tablet_manager.cc:1403] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:50:48.503974 2080 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.504096 2074 ts_tablet_manager.cc:1434] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 14:50:48.504187 2080 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.504678 2073 raft_consensus.cc:359] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.504878 2073 raft_consensus.cc:385] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:48.504962 2073 raft_consensus.cc:740] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 719824530fc143f3975542d473f4d1a8, State: Initialized, Role: FOLLOWER
I20251027 14:50:48.505188 2073 consensus_queue.cc:260] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.505286 2072 tablet_bootstrap.cc:492] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: No bootstrap required, opened a new log
I20251027 14:50:48.505431 2072 ts_tablet_manager.cc:1403] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251027 14:50:48.505584 2079 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.505684 2073 ts_tablet_manager.cc:1434] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 14:50:48.505787 2079 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.506093 2072 raft_consensus.cc:359] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.506245 2072 raft_consensus.cc:385] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:48.506886 2072 raft_consensus.cc:740] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2fb3bb11c72844d98bb162991bda140f, State: Initialized, Role: FOLLOWER
I20251027 14:50:48.507179 2072 consensus_queue.cc:260] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.508252 2072 ts_tablet_manager.cc:1434] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: Time spent starting tablet: real 0.003s user 0.001s sys 0.000s
I20251027 14:50:48.508205 2078 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.508597 2078 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.549363 2079 raft_consensus.cc:493] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:48.549551 2079 raft_consensus.cc:515] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.550139 2079 leader_election.cc:290] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611), 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203)
I20251027 14:50:48.550724 2003 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a671deb6e1514cd0821485590303252b" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3bb11c72844d98bb162991bda140f" is_pre_election: true
I20251027 14:50:48.550714 1862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a671deb6e1514cd0821485590303252b" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e30348760a041b0b51ffed790ca387e" is_pre_election: true
I20251027 14:50:48.550956 2003 raft_consensus.cc:2468] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 0.
I20251027 14:50:48.550940 1862 raft_consensus.cc:2468] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 0.
I20251027 14:50:48.551362 1653 leader_election.cc:304] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 719824530fc143f3975542d473f4d1a8, 8e30348760a041b0b51ffed790ca387e; no voters:
I20251027 14:50:48.551723 2079 raft_consensus.cc:2804] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:48.551829 2079 raft_consensus.cc:493] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:48.551924 2079 raft_consensus.cc:3060] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:48.555124 2079 raft_consensus.cc:515] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.555783 2079 leader_election.cc:290] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 election: Requested vote from peers 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611), 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203)
I20251027 14:50:48.556191 1862 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a671deb6e1514cd0821485590303252b" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8e30348760a041b0b51ffed790ca387e"
I20251027 14:50:48.556424 1862 raft_consensus.cc:3060] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:48.556465 2003 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a671deb6e1514cd0821485590303252b" candidate_uuid: "719824530fc143f3975542d473f4d1a8" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2fb3bb11c72844d98bb162991bda140f"
I20251027 14:50:48.556669 2003 raft_consensus.cc:3060] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:48.558923 1862 raft_consensus.cc:2468] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 1.
I20251027 14:50:48.559293 1653 leader_election.cc:304] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 719824530fc143f3975542d473f4d1a8, 8e30348760a041b0b51ffed790ca387e; no voters:
I20251027 14:50:48.559360 2003 raft_consensus.cc:2468] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 719824530fc143f3975542d473f4d1a8 in term 1.
I20251027 14:50:48.559532 2079 raft_consensus.cc:2804] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:48.559767 2079 raft_consensus.cc:697] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [term 1 LEADER]: Becoming Leader. State: Replica: 719824530fc143f3975542d473f4d1a8, State: Running, Role: LEADER
I20251027 14:50:48.560041 2079 consensus_queue.cc:237] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } }
I20251027 14:50:48.561336 2104 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: TxnStatusTablet state changed. Reason: New leader 719824530fc143f3975542d473f4d1a8. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.561497 2104 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:50:48.562063 2117 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251027 14:50:48.562297 2117 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251027 14:50:48.562981 1561 catalog_manager.cc:5649] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 reported cstate change: term changed from 0 to 1, leader changed from <none> to 719824530fc143f3975542d473f4d1a8 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } health_report { overall_health: UNKNOWN } } }
I20251027 14:50:48.585254 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.579024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36618 (local address 127.0.104.1:33743)
1027 14:50:48.580015 (+ 991us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.580019 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.580034 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:50:48.580108 (+ 74us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.580112 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.580196 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.580322 (+ 126us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:48.581021 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.582050 (+ 1029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.583629 (+ 1579us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.583983 (+ 354us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.584051 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:48.584685 (+ 634us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:48.584820 (+ 135us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.584954 (+ 134us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.585033 (+ 79us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":897,"thread_start_us":770,"threads_started":1}
I20251027 14:50:48.585333 2111 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.579055 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33743 (local address 127.0.0.1:36618)
1027 14:50:48.579280 (+ 225us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:48.579302 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:48.579473 (+ 171us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:48.580369 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:48.580398 (+ 29us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:48.580414 (+ 16us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:48.580883 (+ 469us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.580897 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.582176 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.582182 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:48.583472 (+ 1290us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.583486 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.583631 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.583694 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:48.584812 (+ 1118us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:48.584834 (+ 22us) client_negotiation.cc:769] Sending connection context
1027 14:50:48.584918 (+ 84us) client_negotiation.cc:240] Negotiation successful
1027 14:50:48.585060 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":52}
I20251027 14:50:48.590343 2003 raft_consensus.cc:1275] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Refusing update from remote peer 719824530fc143f3975542d473f4d1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:48.590660 1862 raft_consensus.cc:1275] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Refusing update from remote peer 719824530fc143f3975542d473f4d1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:48.591035 2079 consensus_queue.cc:1048] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:48.591616 2079 consensus_queue.cc:1048] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:48.595005 2080 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: TxnStatusTablet state changed. Reason: New leader 719824530fc143f3975542d473f4d1a8. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.595250 2080 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.600328 2080 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.600577 2080 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 8e30348760a041b0b51ffed790ca387e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.606200 2104 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.606470 2104 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:50:48.611141 2078 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: TxnStatusTablet state changed. Reason: New leader 719824530fc143f3975542d473f4d1a8. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.611397 2078 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.617784 2122 mvcc.cc:204] Tried to move back new op lower bound from 7215417952619499520 to 7215417952504049664. Current Snapshot: MvccSnapshot[applied={T|T < 7215417952619499520}]
I20251027 14:50:48.619674 2079 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.620054 2079 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 719824530fc143f3975542d473f4d1a8: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:50:48.625356 2078 tablet_replica.cc:442] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "719824530fc143f3975542d473f4d1a8" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "719824530fc143f3975542d473f4d1a8" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33743 } } peers { permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 } } peers { permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 } } }
I20251027 14:50:48.625587 2078 tablet_replica.cc:445] T a671deb6e1514cd0821485590303252b P 2fb3bb11c72844d98bb162991bda140f: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:48.628427 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.613409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36620 (local address 127.0.104.1:33743)
1027 14:50:48.613607 (+ 198us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.613614 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.621159 (+ 7545us) server_negotiation.cc:408] Connection header received
1027 14:50:48.621370 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.621377 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.621495 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.621598 (+ 103us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:48.622496 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.624692 (+ 2196us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.626694 (+ 2002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.627150 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.627511 (+ 361us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:48.627666 (+ 155us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:48.627759 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.627972 (+ 213us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.628073 (+ 101us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:50:48.644454 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.638871 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:42855 (local address 127.0.104.1:33743)
1027 14:50:48.639047 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.639054 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.639525 (+ 471us) server_negotiation.cc:408] Connection header received
1027 14:50:48.639708 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.639712 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.639771 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.639847 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:48.640835 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.641753 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.643391 (+ 1638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.643737 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.643786 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:48.643902 (+ 116us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:48.643973 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.644113 (+ 140us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.644186 (+ 73us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:50:48.644600 2134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.638786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33743 (local address 127.0.104.1:42855)
1027 14:50:48.639424 (+ 638us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:48.639443 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:48.639605 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:48.639875 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:48.639881 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:48.639894 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:48.640713 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.640727 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.641915 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.641921 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:48.643264 (+ 1343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:48.643278 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.643459 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.643529 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:48.643995 (+ 466us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:48.644005 (+ 10us) client_negotiation.cc:769] Sending connection context
1027 14:50:48.644084 (+ 79us) client_negotiation.cc:240] Negotiation successful
1027 14:50:48.644216 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":468,"thread_start_us":340,"threads_started":1}
I20251027 14:50:48.654723 2003 raft_consensus.cc:1275] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Refusing update from remote peer 719824530fc143f3975542d473f4d1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:48.654824 1862 raft_consensus.cc:1275] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Refusing update from remote peer 719824530fc143f3975542d473f4d1a8: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:48.655308 2104 consensus_queue.cc:1048] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8e30348760a041b0b51ffed790ca387e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 44611 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:48.655541 2129 consensus_queue.cc:1048] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2fb3bb11c72844d98bb162991bda140f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46203 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:48.719250 1561 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47392:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:48.719504 1561 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47392:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:48.722380 1561 catalog_manager.cc:5936] T 00000000000000000000000000000000 P bf16a95f29724fed95c5103324ab0a12: Sending DeleteTablet for 3 replicas of tablet 166ec1eb718344e3b7a9b4fa3ce38a13
I20251027 14:50:48.723239 1700 tablet_service.cc:1552] Processing DeleteTablet for tablet 166ec1eb718344e3b7a9b4fa3ce38a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:36612
I20251027 14:50:48.723598 1842 tablet_service.cc:1552] Processing DeleteTablet for tablet 166ec1eb718344e3b7a9b4fa3ce38a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:35200
I20251027 14:50:48.723894 1982 tablet_service.cc:1552] Processing DeleteTablet for tablet 166ec1eb718344e3b7a9b4fa3ce38a13 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:48 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40276
I20251027 14:50:48.734239 2138 tablet_replica.cc:333] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: stopping tablet replica
I20251027 14:50:48.734592 2138 raft_consensus.cc:2243] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:50:48.735270 2138 raft_consensus.cc:2272] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:48.736552 2138 ts_tablet_manager.cc:1916] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:48.739533 2139 tablet_replica.cc:333] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: stopping tablet replica
I20251027 14:50:48.739873 2139 raft_consensus.cc:2243] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:48.740176 2139 raft_consensus.cc:2272] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:48.741182 2140 tablet_replica.cc:333] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: stopping tablet replica
I20251027 14:50:48.741425 2139 ts_tablet_manager.cc:1916] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:48.741474 2140 raft_consensus.cc:2243] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:48.741796 2140 raft_consensus.cc:2272] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:48.744786 2138 ts_tablet_manager.cc:1929] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:48.744969 2138 log.cc:1199] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/166ec1eb718344e3b7a9b4fa3ce38a13
I20251027 14:50:48.745918 2138 ts_tablet_manager.cc:1950] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 719824530fc143f3975542d473f4d1a8: Deleting consensus metadata
I20251027 14:50:48.747071 1549 catalog_manager.cc:4985] TS 719824530fc143f3975542d473f4d1a8 (127.0.104.1:33743): tablet 166ec1eb718344e3b7a9b4fa3ce38a13 (table test-table [id=b16a569f40764cb9b236f0b9768ebbe1]) successfully deleted
I20251027 14:50:48.749095 2140 ts_tablet_manager.cc:1916] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:48.749647 2139 ts_tablet_manager.cc:1929] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:48.750614 2139 log.cc:1199] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/166ec1eb718344e3b7a9b4fa3ce38a13
I20251027 14:50:48.752817 2139 ts_tablet_manager.cc:1950] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 8e30348760a041b0b51ffed790ca387e: Deleting consensus metadata
I20251027 14:50:48.753947 1548 catalog_manager.cc:4985] TS 8e30348760a041b0b51ffed790ca387e (127.0.104.2:44611): tablet 166ec1eb718344e3b7a9b4fa3ce38a13 (table test-table [id=b16a569f40764cb9b236f0b9768ebbe1]) successfully deleted
I20251027 14:50:48.757114 2140 ts_tablet_manager.cc:1929] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:48.757233 2140 log.cc:1199] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SmokeTestAsAuthorizedUser.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/166ec1eb718344e3b7a9b4fa3ce38a13
I20251027 14:50:48.757710 2140 ts_tablet_manager.cc:1950] T 166ec1eb718344e3b7a9b4fa3ce38a13 P 2fb3bb11c72844d98bb162991bda140f: Deleting consensus metadata
Oct 27 14:50:48 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:48.758716 1548 catalog_manager.cc:4985] TS 2fb3bb11c72844d98bb162991bda140f (127.0.104.3:46203): tablet 166ec1eb718344e3b7a9b4fa3ce38a13 (table test-table [id=b16a569f40764cb9b236f0b9768ebbe1]) successfully deleted
I20251027 14:50:48.764128 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.737660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36624 (local address 127.0.104.1:33743)
1027 14:50:48.737841 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.737847 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.750377 (+ 12530us) server_negotiation.cc:408] Connection header received
1027 14:50:48.750566 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.750590 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.750700 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.750813 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:48.752476 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.753816 (+ 1340us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.755099 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.755907 (+ 808us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.758694 (+ 2787us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:48.758757 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:48.758772 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:48.758806 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:48.761831 (+ 3025us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.762526 (+ 695us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.762534 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.762536 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.762609 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.762909 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.762912 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.762914 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.763170 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:48.763414 (+ 244us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.763661 (+ 247us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.763830 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
W20251027 14:50:48.764822 1766 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.SetFlag from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36624
I20251027 14:50:48.780215 2112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.767837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47404 (local address 127.0.104.62:37547)
1027 14:50:48.768014 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.768018 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.768308 (+ 290us) server_negotiation.cc:408] Connection header received
1027 14:50:48.768494 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.768499 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.768562 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.768637 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:48.770547 (+ 1910us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.771484 (+ 937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.772926 (+ 1442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.773346 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.775374 (+ 2028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:48.775404 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:48.775408 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:48.775435 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:48.777841 (+ 2406us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.778488 (+ 647us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.778493 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.778495 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.778576 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.778840 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.778843 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.778845 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.779038 (+ 193us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:48.779207 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.779471 (+ 264us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.779660 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
W20251027 14:50:48.780977 1561 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:47404
I20251027 14:50:48.783005 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1635
I20251027 14:50:48.796890 2120 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.784667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36632 (local address 127.0.104.1:33743)
1027 14:50:48.784842 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.784848 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.785141 (+ 293us) server_negotiation.cc:408] Connection header received
1027 14:50:48.785275 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.785281 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.785365 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.785468 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:48.786815 (+ 1347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.788286 (+ 1471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.789361 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.789815 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.791197 (+ 1382us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:48.791235 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:48.791241 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:48.791281 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:48.794603 (+ 3322us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.795125 (+ 522us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.795131 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.795135 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.795215 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.795538 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.795543 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.795547 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.795827 (+ 280us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:48.796079 (+ 252us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.796274 (+ 195us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.796525 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251027 14:50:48.797892 1766 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:36632
W20251027 14:50:48.798439 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1635: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:50:48.798612 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1635
I20251027 14:50:48.814098 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1777
Oct 27 14:50:48 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:50:48.827282 2071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.814750 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35218 (local address 127.0.104.2:44611)
1027 14:50:48.814908 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.814912 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.814928 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:50:48.815088 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.815093 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.815154 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.815239 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:48.816441 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.817272 (+ 831us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.818264 (+ 992us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.818615 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.821926 (+ 3311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:48.821970 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:48.821986 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:48.822021 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:48.824827 (+ 2806us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.825482 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.825490 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.825492 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.825596 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.825884 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.825889 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.825891 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.826201 (+ 310us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:48.826474 (+ 273us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.826721 (+ 247us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.826949 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20251027 14:50:48.827879 1907 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:35218
W20251027 14:50:48.828459 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1777: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:50:48.828610 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1777
I20251027 14:50:48.845605 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1918
Oct 27 14:50:48 dist-test-slave-hxxt krb5kdc[1514](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576647, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:50:48.859014 2067 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:48.846381 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40286 (local address 127.0.104.3:46203)
1027 14:50:48.846534 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:48.846539 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:48.846555 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:50:48.846680 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:48.846685 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:48.846753 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:48.846833 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:48.848133 (+ 1300us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.849031 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:48.849974 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:48.850347 (+ 373us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:48.854067 (+ 3720us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:48.854103 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:48.854119 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:48.854152 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:48.856964 (+ 2812us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.857559 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.857568 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.857571 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.857664 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:48.857895 (+ 231us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:48.857897 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:48.857899 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:48.858160 (+ 261us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:48.858395 (+ 235us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:48.858538 (+ 143us) server_negotiation.cc:300] Negotiation successful
1027 14:50:48.858709 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
W20251027 14:50:48.859668 2048 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40286
W20251027 14:50:48.860278 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1918: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:50:48.860423 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1918
I20251027 14:50:48.873590 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 1530
I20251027 14:50:48.874635 1602 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:48.984802 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 1530
2025-10-27T14:50:48Z chronyd exiting
[ OK ] SecurityITest.SmokeTestAsAuthorizedUser (5520 ms)
[ RUN ] SecurityITest.TxnSmokeWithDifferentUserTypes
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:50:49 dist-test-slave-hxxt krb5kdc[2149](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:50:49 dist-test-slave-hxxt krb5kdc[2149](info): set up 2 sockets
Oct 27 14:50:49 dist-test-slave-hxxt krb5kdc[2149](info): commencing operation
krb5kdc: starting...
W20251027 14:50:50.864302 416 mini_kdc.cc:121] Time spent starting KDC: real 1.842s user 0.000s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:50:50 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:50:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:51.058738 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46743
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:46743
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:51.369669 2165 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:51.369992 2165 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:51.370117 2165 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:51.378068 2165 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:50:51.378176 2165 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251027 14:50:51.378250 2165 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:51.378319 2165 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:51.378404 2165 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:50:51.378464 2165 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:50:51.388859 2165 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:46743
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46743
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:51.390431 2165 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:51.391716 2165 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:51.399744 2171 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:51.399770 2170 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:51.399998 2173 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:51.400671 2165 server_base.cc:1047] running on GCE node
I20251027 14:50:51.401343 2165 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:51.402626 2165 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:51.403813 2165 hybrid_clock.cc:648] HybridClock initialized: now 1761576651403762 us; error 49 us; skew 500 ppm
I20251027 14:50:51.404110 2165 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:51 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:51.406955 2165 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:50:51.408350 2165 webserver.cc:492] Webserver started at http://127.0.104.62:36175/ using document root <none> and password file <none>
I20251027 14:50:51.409017 2165 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:51.409102 2165 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:51.409395 2165 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:51.411785 2165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "4d74a3c2ce28416b880846cb7275ab04"
format_stamp: "Formatted at 2025-10-27 14:50:51 on dist-test-slave-hxxt"
server_key: "7f792889b50bf3879ee53ccc857d2e51"
server_key_iv: "1029f41f9f637583a4f3537ec2297087"
server_key_version: "encryptionkey@0"
I20251027 14:50:51.412495 2165 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "4d74a3c2ce28416b880846cb7275ab04"
format_stamp: "Formatted at 2025-10-27 14:50:51 on dist-test-slave-hxxt"
server_key: "7f792889b50bf3879ee53ccc857d2e51"
server_key_iv: "1029f41f9f637583a4f3537ec2297087"
server_key_version: "encryptionkey@0"
I20251027 14:50:51.417527 2165 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:50:51.420715 2180 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:51.422001 2165 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:50:51.422137 2165 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "4d74a3c2ce28416b880846cb7275ab04"
format_stamp: "Formatted at 2025-10-27 14:50:51 on dist-test-slave-hxxt"
server_key: "7f792889b50bf3879ee53ccc857d2e51"
server_key_iv: "1029f41f9f637583a4f3537ec2297087"
server_key_version: "encryptionkey@0"
I20251027 14:50:51.422292 2165 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:51.462183 2165 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:51.469323 2165 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:51.469560 2165 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:51.489848 2165 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46743
I20251027 14:50:51.489871 2241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46743 every 8 connection(s)
I20251027 14:50:51.491276 2165 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:50:51.493930 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 2165
I20251027 14:50:51.494092 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:50:51.494661 416 external_mini_cluster.cc:1468] Setting key 555302a39f21d9adb4cf16e6af57047b
I20251027 14:50:51.495854 2242 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 27 14:50:51 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576650, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:51.508493 2242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Bootstrap starting.
I20251027 14:50:51.511691 2242 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:51.512936 2242 log.cc:826] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:51.515115 2245 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:51.496580 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53746 (local address 127.0.104.62:46743)
1027 14:50:51.497333 (+ 753us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:51.497349 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:51.497396 (+ 47us) server_negotiation.cc:408] Connection header received
1027 14:50:51.499003 (+ 1607us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:51.499032 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:51.499577 (+ 545us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:51.500121 (+ 544us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:51.501547 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:51.503519 (+ 1972us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:51.504694 (+ 1175us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:51.505290 (+ 596us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:51.508268 (+ 2978us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:51.508360 (+ 92us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:51.508398 (+ 38us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:51.508439 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:51.511161 (+ 2722us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:51.511946 (+ 785us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:51.511953 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:51.511963 (+ 10us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:51.512063 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:51.512486 (+ 423us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:51.512491 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:51.512493 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:51.512985 (+ 492us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:51.513339 (+ 354us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:51.513733 (+ 394us) server_negotiation.cc:300] Negotiation successful
1027 14:50:51.514149 (+ 416us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":503,"thread_start_us":323,"threads_started":1}
I20251027 14:50:51.515901 2242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: No bootstrap required, opened a new log
I20251027 14:50:51.521466 2242 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74a3c2ce28416b880846cb7275ab04" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46743 } }
I20251027 14:50:51.521836 2242 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:51.521932 2242 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d74a3c2ce28416b880846cb7275ab04, State: Initialized, Role: FOLLOWER
I20251027 14:50:51.522518 2242 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74a3c2ce28416b880846cb7275ab04" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46743 } }
I20251027 14:50:51.522708 2242 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:50:51.522820 2242 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:50:51.522964 2242 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:51.525563 2242 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74a3c2ce28416b880846cb7275ab04" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46743 } }
I20251027 14:50:51.526067 2242 leader_election.cc:304] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4d74a3c2ce28416b880846cb7275ab04; no voters:
I20251027 14:50:51.526669 2242 leader_election.cc:290] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:50:51.526753 2247 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:51.527403 2247 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [term 1 LEADER]: Becoming Leader. State: Replica: 4d74a3c2ce28416b880846cb7275ab04, State: Running, Role: LEADER
I20251027 14:50:51.527856 2247 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74a3c2ce28416b880846cb7275ab04" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46743 } }
I20251027 14:50:51.528678 2242 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:50:51.530447 2249 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d74a3c2ce28416b880846cb7275ab04. Latest consensus state: current_term: 1 leader_uuid: "4d74a3c2ce28416b880846cb7275ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74a3c2ce28416b880846cb7275ab04" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46743 } } }
I20251027 14:50:51.530791 2249 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:51.536028 2248 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4d74a3c2ce28416b880846cb7275ab04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4d74a3c2ce28416b880846cb7275ab04" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46743 } } }
I20251027 14:50:51.536783 2248 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04 [sys.catalog]: This master's current role is: LEADER
I20251027 14:50:51.537240 2255 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:50:51.543128 2255 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:50:51.551496 2255 catalog_manager.cc:1357] Generated new cluster ID: c105aac1d05a48f89243929443e0bfdc
I20251027 14:50:51.551630 2255 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:50:51.561873 2255 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:50:51.563485 2255 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:50:51.578234 2255 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Generated new TSK 0
I20251027 14:50:51.579291 2255 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:51.678016 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46743
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:52.033706 2270 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:52.033995 2270 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:52.034132 2270 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:52.042398 2270 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:52.042497 2270 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:52.042641 2270 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251027 14:50:52.042688 2270 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:50:52.042814 2270 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:50:52.053171 2270 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:52.054754 2270 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:52.056143 2270 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:52.065079 2275 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:52.065157 2278 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:52.065331 2276 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:52.066568 2270 server_base.cc:1047] running on GCE node
I20251027 14:50:52.067092 2270 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:52.067921 2270 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:52.069156 2270 hybrid_clock.cc:648] HybridClock initialized: now 1761576652069084 us; error 44 us; skew 500 ppm
I20251027 14:50:52.069448 2270 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:52 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:52.072309 2270 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:50:52.073745 2270 webserver.cc:492] Webserver started at http://127.0.104.1:37641/ using document root <none> and password file <none>
I20251027 14:50:52.074424 2270 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:52.074501 2270 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:52.074800 2270 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:52.077324 2270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "b02454102e084805bdbebeab42c0538e"
format_stamp: "Formatted at 2025-10-27 14:50:52 on dist-test-slave-hxxt"
server_key: "efce864ebb382b4b0744ce831708a46d"
server_key_iv: "b77173bc65757cbd89038c326064d0e2"
server_key_version: "encryptionkey@0"
I20251027 14:50:52.078039 2270 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "b02454102e084805bdbebeab42c0538e"
format_stamp: "Formatted at 2025-10-27 14:50:52 on dist-test-slave-hxxt"
server_key: "efce864ebb382b4b0744ce831708a46d"
server_key_iv: "b77173bc65757cbd89038c326064d0e2"
server_key_version: "encryptionkey@0"
I20251027 14:50:52.082871 2270 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:50:52.085963 2285 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:52.087148 2270 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20251027 14:50:52.087289 2270 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "b02454102e084805bdbebeab42c0538e"
format_stamp: "Formatted at 2025-10-27 14:50:52 on dist-test-slave-hxxt"
server_key: "efce864ebb382b4b0744ce831708a46d"
server_key_iv: "b77173bc65757cbd89038c326064d0e2"
server_key_version: "encryptionkey@0"
I20251027 14:50:52.087466 2270 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:52.115939 2270 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:52.119994 2270 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:52.120241 2270 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:52.123032 2270 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:52.123131 2270 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:52.123265 2270 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:52.123349 2270 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:50:52 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:52.142834 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.123489 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:46575 (local address 127.0.104.62:46743)
1027 14:50:52.124180 (+ 691us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:52.124184 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:52.125201 (+ 1017us) server_negotiation.cc:408] Connection header received
1027 14:50:52.126996 (+ 1795us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:52.127003 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:52.127090 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:52.127195 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:52.129456 (+ 2261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.130662 (+ 1206us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.132216 (+ 1554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.132692 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.136430 (+ 3738us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:52.136471 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:52.136476 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:52.136516 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:52.139789 (+ 3273us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.140577 (+ 788us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.140584 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.140588 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.140716 (+ 128us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.141015 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.141021 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.141024 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.141325 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:52.141476 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:52.142162 (+ 686us) server_negotiation.cc:300] Negotiation successful
1027 14:50:52.142434 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":593,"thread_start_us":505,"threads_started":1}
I20251027 14:50:52.143630 2295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.124177 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.104.1:46575)
1027 14:50:52.125045 (+ 868us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:52.125088 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:52.126701 (+ 1613us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:52.127518 (+ 817us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:52.127528 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:52.128166 (+ 638us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:52.129206 (+ 1040us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.129230 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.130820 (+ 1590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.130829 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:52.132063 (+ 1234us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.132078 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.132451 (+ 373us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.133365 (+ 914us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:52.133390 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:52.136191 (+ 2801us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:52.139933 (+ 3742us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.139940 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.139955 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.140458 (+ 503us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.140804 (+ 346us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.140808 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.140810 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.140925 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.141427 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:52.141433 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:52.141776 (+ 343us) client_negotiation.cc:769] Sending connection context
1027 14:50:52.142055 (+ 279us) client_negotiation.cc:240] Negotiation successful
1027 14:50:52.142431 (+ 376us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":623,"thread_start_us":425,"threads_started":1}
I20251027 14:50:52.162921 2363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.150785 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.104.1:42087)
1027 14:50:52.151336 (+ 551us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:52.151356 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:52.151485 (+ 129us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:52.151780 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:52.151785 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:52.152243 (+ 458us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:52.152857 (+ 614us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.152872 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.154397 (+ 1525us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.154402 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:52.155697 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.155711 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.155925 (+ 214us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.156586 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:52.156604 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:52.157243 (+ 639us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:52.160801 (+ 3558us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.160805 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.160808 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.161153 (+ 345us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.161419 (+ 266us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.161422 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.161424 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.161484 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.161970 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:52.161972 (+ 2us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:52.162093 (+ 121us) client_negotiation.cc:769] Sending connection context
1027 14:50:52.162212 (+ 119us) client_negotiation.cc:240] Negotiation successful
1027 14:50:52.162516 (+ 304us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":441,"thread_start_us":320,"threads_started":1}
I20251027 14:50:52.162933 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.150691 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:42087 (local address 127.0.104.62:46743)
1027 14:50:52.150833 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:52.150837 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:52.151440 (+ 603us) server_negotiation.cc:408] Connection header received
1027 14:50:52.151601 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:52.151606 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:52.151665 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:52.151743 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:52.153051 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.154169 (+ 1118us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.155856 (+ 1687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.156354 (+ 498us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.157385 (+ 1031us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:52.157429 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:52.157435 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:52.157482 (+ 47us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:52.160675 (+ 3193us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.161249 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.161255 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.161259 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.161339 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.161555 (+ 216us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.161560 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.161563 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.161882 (+ 319us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:52.162018 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:52.162345 (+ 327us) server_negotiation.cc:300] Negotiation successful
1027 14:50:52.162613 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:50:52.171464 2270 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37543
I20251027 14:50:52.171502 2405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37543 every 8 connection(s)
I20251027 14:50:52.172811 2270 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:50:52.175223 2406 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46743
I20251027 14:50:52.175485 2406 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:52.176235 2406 heartbeater.cc:507] Master 127.0.104.62:46743 requested a full tablet report, sending...
I20251027 14:50:52.177716 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 2270
I20251027 14:50:52.177886 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:50:52.178369 2197 ts_manager.cc:194] Registered new tserver with Master: b02454102e084805bdbebeab42c0538e (127.0.104.1:37543)
I20251027 14:50:52.178429 416 external_mini_cluster.cc:1468] Setting key c5e4ac64911201612d6ee4a93d228e47
I20251027 14:50:52.180051 2197 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:46575
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:52.246012 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46743
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:52.568053 2412 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:52.568331 2412 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:52.568507 2412 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:52.576458 2412 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:52.576553 2412 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:52.576756 2412 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251027 14:50:52.576809 2412 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:50:52.576931 2412 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:50:52.587028 2412 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:52.588630 2412 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:52.589907 2412 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:52.599198 2417 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:52.599200 2418 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:52.599463 2420 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:52.600067 2412 server_base.cc:1047] running on GCE node
I20251027 14:50:52.600658 2412 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:52.601523 2412 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:52.602737 2412 hybrid_clock.cc:648] HybridClock initialized: now 1761576652602665 us; error 46 us; skew 500 ppm
I20251027 14:50:52.603001 2412 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:52 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:52.605973 2412 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:50:52.607432 2412 webserver.cc:492] Webserver started at http://127.0.104.2:35859/ using document root <none> and password file <none>
I20251027 14:50:52.608145 2412 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:52.608239 2412 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:52.608626 2412 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:52.611119 2412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "4b59f5c96e6d453999a2713bde3dd5b2"
format_stamp: "Formatted at 2025-10-27 14:50:52 on dist-test-slave-hxxt"
server_key: "e5a5d23fe55d37f2f46689c51bb469b2"
server_key_iv: "46500d306dbd4007575b73454a1563ea"
server_key_version: "encryptionkey@0"
I20251027 14:50:52.611850 2412 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "4b59f5c96e6d453999a2713bde3dd5b2"
format_stamp: "Formatted at 2025-10-27 14:50:52 on dist-test-slave-hxxt"
server_key: "e5a5d23fe55d37f2f46689c51bb469b2"
server_key_iv: "46500d306dbd4007575b73454a1563ea"
server_key_version: "encryptionkey@0"
I20251027 14:50:52.616902 2412 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251027 14:50:52.620029 2427 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:52.621335 2412 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20251027 14:50:52.621481 2412 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "4b59f5c96e6d453999a2713bde3dd5b2"
format_stamp: "Formatted at 2025-10-27 14:50:52 on dist-test-slave-hxxt"
server_key: "e5a5d23fe55d37f2f46689c51bb469b2"
server_key_iv: "46500d306dbd4007575b73454a1563ea"
server_key_version: "encryptionkey@0"
I20251027 14:50:52.621665 2412 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:52.653774 2412 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:52.657469 2412 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:52.657711 2412 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:52.660301 2412 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:52.660405 2412 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:52.660528 2412 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:52.660584 2412 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:50:52 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576652, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:52.680115 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.660913 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:55879 (local address 127.0.104.62:46743)
1027 14:50:52.661089 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:52.661096 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:52.662491 (+ 1395us) server_negotiation.cc:408] Connection header received
1027 14:50:52.664452 (+ 1961us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:52.664459 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:52.664602 (+ 143us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:52.664758 (+ 156us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:52.667250 (+ 2492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.668591 (+ 1341us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.670201 (+ 1610us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.670663 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.673866 (+ 3203us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:52.673899 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:52.673902 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:52.673931 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:52.676968 (+ 3037us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.677841 (+ 873us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.677849 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.677853 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.677946 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.678278 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.678284 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.678287 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.678622 (+ 335us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:52.678766 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:52.679453 (+ 687us) server_negotiation.cc:300] Negotiation successful
1027 14:50:52.679732 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251027 14:50:52.680992 2436 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.661501 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.104.2:55879)
1027 14:50:52.662304 (+ 803us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:52.662355 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:52.663962 (+ 1607us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:52.665195 (+ 1233us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:52.665206 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:52.666042 (+ 836us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:52.667058 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.667082 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.668765 (+ 1683us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.668771 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:52.669991 (+ 1220us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.670010 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.670312 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.670993 (+ 681us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:52.671019 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:52.673665 (+ 2646us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:52.677116 (+ 3451us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.677125 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.677147 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.677702 (+ 555us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.678071 (+ 369us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.678075 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.678077 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.678186 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.678730 (+ 544us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:52.678736 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:52.679043 (+ 307us) client_negotiation.cc:769] Sending connection context
1027 14:50:52.679336 (+ 293us) client_negotiation.cc:240] Negotiation successful
1027 14:50:52.679687 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":541,"thread_start_us":365,"threads_started":1}
I20251027 14:50:52.700320 2490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.686256 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.104.2:37891)
1027 14:50:52.686815 (+ 559us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:52.686831 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:52.686956 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:52.687228 (+ 272us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:52.687234 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:52.687721 (+ 487us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:52.688412 (+ 691us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.688429 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.690094 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.690100 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:52.691286 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:52.691297 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.691492 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.692264 (+ 772us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:52.692281 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:52.693124 (+ 843us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:52.696813 (+ 3689us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.696817 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.696821 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.697232 (+ 411us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.697649 (+ 417us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:52.697654 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:52.697658 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:52.697750 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:52.699404 (+ 1654us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:52.699414 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:52.699585 (+ 171us) client_negotiation.cc:769] Sending connection context
1027 14:50:52.699682 (+ 97us) client_negotiation.cc:240] Negotiation successful
1027 14:50:52.699888 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":451,"thread_start_us":332,"threads_started":1}
I20251027 14:50:52.701001 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:52.686178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:37891 (local address 127.0.104.62:46743)
1027 14:50:52.686307 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:52.686311 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:52.686917 (+ 606us) server_negotiation.cc:408] Connection header received
1027 14:50:52.687057 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:52.687061 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:52.687114 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:52.687188 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:52.688628 (+ 1440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.689955 (+ 1327us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:52.691412 (+ 1457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:52.691838 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:52.693275 (+ 1437us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:52.693315 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:52.693319 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:52.693346 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:52.696667 (+ 3321us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.697377 (+ 710us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.697383 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.697386 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.697497 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:52.698857 (+ 1360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:52.698863 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:52.698866 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:52.699220 (+ 354us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:52.700443 (+ 1223us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:52.700569 (+ 126us) server_negotiation.cc:300] Negotiation successful
1027 14:50:52.700798 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251027 14:50:52.708855 2412 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:36959
I20251027 14:50:52.708886 2546 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:36959 every 8 connection(s)
I20251027 14:50:52.710098 2412 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:50:52.712018 2547 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46743
I20251027 14:50:52.712286 2547 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:52.712689 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 2412
I20251027 14:50:52.712843 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:50:52.713081 2547 heartbeater.cc:507] Master 127.0.104.62:46743 requested a full tablet report, sending...
I20251027 14:50:52.713227 416 external_mini_cluster.cc:1468] Setting key cf8ff815cf771dd8de4ca3ef319e4398
I20251027 14:50:52.714531 2196 ts_manager.cc:194] Registered new tserver with Master: 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959)
I20251027 14:50:52.715579 2196 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:55879
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:50:52.781639 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46743
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33019
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true
--txn_keepalive_interval_ms=500 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:50:53.092080 2553 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:50:53.092428 2553 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:50:53.092568 2553 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:50:53.100591 2553 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:50:53.100708 2553 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:50:53.100849 2553 flags.cc:425] Enabled experimental flag: --txn_keepalive_interval_ms=500
W20251027 14:50:53.100901 2553 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:50:53.101017 2553 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:50:53.111264 2553 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33019
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--txn_keepalive_interval_ms=500
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:46743
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:50:53.112907 2553 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:50:53.114249 2553 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:50:53.123185 2559 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:53.123365 2558 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:50:53.123754 2561 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:50:53.124262 2553 server_base.cc:1047] running on GCE node
I20251027 14:50:53.124903 2553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:50:53.125851 2553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:50:53.127067 2553 hybrid_clock.cc:648] HybridClock initialized: now 1761576653126984 us; error 61 us; skew 500 ppm
I20251027 14:50:53.127333 2553 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:50:53 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:50:53.130532 2553 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:50:53.132105 2553 webserver.cc:492] Webserver started at http://127.0.104.3:44043/ using document root <none> and password file <none>
I20251027 14:50:53.132922 2553 fs_manager.cc:362] Metadata directory not provided
I20251027 14:50:53.133011 2553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:50:53.133332 2553 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:50:53.136125 2553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "0d3261285c7144e3aa5bd41920599a35"
format_stamp: "Formatted at 2025-10-27 14:50:53 on dist-test-slave-hxxt"
server_key: "29448b13f2eb6733539008281145c649"
server_key_iv: "a6552081349c822d89ca6c902aac8f28"
server_key_version: "encryptionkey@0"
I20251027 14:50:53.136965 2553 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "0d3261285c7144e3aa5bd41920599a35"
format_stamp: "Formatted at 2025-10-27 14:50:53 on dist-test-slave-hxxt"
server_key: "29448b13f2eb6733539008281145c649"
server_key_iv: "a6552081349c822d89ca6c902aac8f28"
server_key_version: "encryptionkey@0"
I20251027 14:50:53.142436 2553 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:50:53.145771 2568 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:53.147107 2553 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:50:53.147258 2553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "0d3261285c7144e3aa5bd41920599a35"
format_stamp: "Formatted at 2025-10-27 14:50:53 on dist-test-slave-hxxt"
server_key: "29448b13f2eb6733539008281145c649"
server_key_iv: "a6552081349c822d89ca6c902aac8f28"
server_key_version: "encryptionkey@0"
I20251027 14:50:53.147454 2553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:50:53.159665 2553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:50:53.163599 2553 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:50:53.163846 2553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:50:53.166543 2553 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:50:53.166635 2553 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:50:53.166745 2553 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:50:53.166798 2553 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:50:53 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576653, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:53.183540 2406 heartbeater.cc:499] Master 127.0.104.62:46743 was elected leader, sending a full tablet report...
I20251027 14:50:53.187397 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.167079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:52197 (local address 127.0.104.62:46743)
1027 14:50:53.167255 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.167262 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.168647 (+ 1385us) server_negotiation.cc:408] Connection header received
1027 14:50:53.170375 (+ 1728us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.170383 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.170477 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.170602 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:53.173072 (+ 2470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.174522 (+ 1450us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.176557 (+ 2035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.177039 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.180465 (+ 3426us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:53.180497 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:53.180501 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:53.180532 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:53.183797 (+ 3265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.184832 (+ 1035us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.184838 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.184841 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.184916 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.185368 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.185373 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.185376 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.185611 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:53.185761 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.186723 (+ 962us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.187006 (+ 283us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:50:53.188205 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.167641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.104.3:52197)
1027 14:50:53.168452 (+ 811us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.168505 (+ 53us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.170032 (+ 1527us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.170802 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.170810 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.171628 (+ 818us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:53.172777 (+ 1149us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.172811 (+ 34us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.174769 (+ 1958us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.174776 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.176317 (+ 1541us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.176334 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.176685 (+ 351us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.177588 (+ 903us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:53.177630 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:53.180217 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:53.183939 (+ 3722us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:53.183946 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:53.183961 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:53.184666 (+ 705us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:53.185038 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:53.185044 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:53.185048 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:53.185223 (+ 175us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:53.185730 (+ 507us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:53.185738 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:53.186165 (+ 427us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.186595 (+ 430us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.187049 (+ 454us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":352,"threads_started":1}
I20251027 14:50:53.211840 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.193471 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:49139 (local address 127.0.104.62:46743)
1027 14:50:53.193638 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.193644 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.197018 (+ 3374us) server_negotiation.cc:408] Connection header received
1027 14:50:53.197249 (+ 231us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.197256 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.197342 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.197460 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:53.199411 (+ 1951us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.200796 (+ 1385us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.203104 (+ 2308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.203624 (+ 520us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.205285 (+ 1661us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:53.205328 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:53.205334 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:53.205374 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:53.208744 (+ 3370us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.209634 (+ 890us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.209642 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.209647 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.209747 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.210117 (+ 370us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.210123 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.210127 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.210540 (+ 413us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:53.210700 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.211194 (+ 494us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.211457 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:50:53.211972 2631 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.193841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.104.3:49139)
1027 14:50:53.196888 (+ 3047us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.196914 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.197080 (+ 166us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.197655 (+ 575us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.197661 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.198392 (+ 731us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:53.199219 (+ 827us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.199236 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.200995 (+ 1759us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.201002 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.202931 (+ 1929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.202951 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.203175 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.203915 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:53.203942 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:53.205109 (+ 1167us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:53.208894 (+ 3785us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:53.208900 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:53.208909 (+ 9us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:53.209491 (+ 582us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:53.209867 (+ 376us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:53.209873 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:53.209877 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:53.209990 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:53.210684 (+ 694us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:53.210690 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:53.210887 (+ 197us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.211056 (+ 169us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.211505 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2878,"thread_start_us":2768,"threads_started":1}
I20251027 14:50:53.214200 2553 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:43281
I20251027 14:50:53.214267 2687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:43281 every 8 connection(s)
I20251027 14:50:53.215505 2553 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:50:53.216670 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 2553
I20251027 14:50:53.216843 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:50:53.217237 416 external_mini_cluster.cc:1468] Setting key 036ea139d8c14d1979ba22023b6fec63
I20251027 14:50:53.217900 2688 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46743
I20251027 14:50:53.218178 2688 heartbeater.cc:461] Registering TS with master...
I20251027 14:50:53.219033 2688 heartbeater.cc:507] Master 127.0.104.62:46743 requested a full tablet report, sending...
I20251027 14:50:53.220248 2196 ts_manager.cc:194] Registered new tserver with Master: 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:53.221047 2196 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:52197
I20251027 14:50:53.232029 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 27 14:50:53 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 27 14:50:53 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576653, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:53.273191 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.261461 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53750 (local address 127.0.104.62:46743)
1027 14:50:53.261641 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.261647 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.261722 (+ 75us) server_negotiation.cc:408] Connection header received
1027 14:50:53.261908 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.261913 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.261975 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.262057 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:53.263550 (+ 1493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.264480 (+ 930us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.265767 (+ 1287us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.266192 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.269161 (+ 2969us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:53.269198 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:53.269202 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:53.269232 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:53.271401 (+ 2169us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.271905 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.271908 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.271911 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.271974 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.272158 (+ 184us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.272161 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.272162 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.272368 (+ 206us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:53.272578 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.272771 (+ 193us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.272944 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251027 14:50:53.276708 2196 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:50:53.281152 2196 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:53.300750 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.292928 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:43281 (local address 127.0.0.1:40844)
1027 14:50:53.293864 (+ 936us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.293893 (+ 29us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.294025 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.295107 (+ 1082us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.295114 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.295143 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.295928 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.295944 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.298238 (+ 2294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.298244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.299823 (+ 1579us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.299834 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.299982 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.300034 (+ 52us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.300183 (+ 149us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.300361 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":753,"thread_start_us":613,"threads_started":1}
I20251027 14:50:53.301749 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.293094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40844 (local address 127.0.104.3:43281)
1027 14:50:53.294622 (+ 1528us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.294629 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.294653 (+ 24us) server_negotiation.cc:408] Connection header received
1027 14:50:53.294726 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.294731 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.294918 (+ 187us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.295116 (+ 198us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.296152 (+ 1036us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.298042 (+ 1890us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.299960 (+ 1918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.301087 (+ 1127us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.301242 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.301351 (+ 109us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.301498 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1352,"thread_start_us":1168,"threads_started":1}
I20251027 14:50:53.303930 2703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.294444 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.0.1:40674)
1027 14:50:53.296361 (+ 1917us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.296410 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.296519 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.297250 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.297256 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.297279 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.297801 (+ 522us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.297809 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.299765 (+ 1956us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.299774 (+ 9us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.303191 (+ 3417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.303207 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.303388 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.303421 (+ 33us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.303487 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.303610 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1803,"thread_start_us":1702,"threads_started":1}
I20251027 14:50:53.305053 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.294739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.104.2:36959)
1027 14:50:53.295781 (+ 1042us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.295790 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.296622 (+ 832us) server_negotiation.cc:408] Connection header received
1027 14:50:53.296708 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.296714 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.296872 (+ 158us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.297054 (+ 182us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.297937 (+ 883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.299538 (+ 1601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.303361 (+ 3823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.304430 (+ 1069us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.304562 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.304646 (+ 84us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.304809 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":912,"thread_start_us":814,"threads_started":1}
I20251027 14:50:53.305964 2701 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.296502 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37543 (local address 127.0.0.1:32768)
1027 14:50:53.300944 (+ 4442us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.300961 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.301056 (+ 95us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.301909 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.301914 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.301935 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.302278 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.302288 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.303805 (+ 1517us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.303811 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.305396 (+ 1585us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.305406 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.305518 (+ 112us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.305539 (+ 21us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.305597 (+ 58us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.305702 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4367,"threads_started":1}
I20251027 14:50:53.306985 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.296741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32768 (local address 127.0.104.1:37543)
1027 14:50:53.297260 (+ 519us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.297268 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.301165 (+ 3897us) server_negotiation.cc:408] Connection header received
1027 14:50:53.301263 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.301270 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.301511 (+ 241us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.301688 (+ 177us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.302433 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.303633 (+ 1200us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.305528 (+ 1895us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.306535 (+ 1007us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.306620 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.306717 (+ 97us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.306825 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":369,"thread_start_us":266,"threads_started":1}
I20251027 14:50:53.312345 2336 tablet_service.cc:1505] Processing CreateTablet for tablet 7e63afde536d48de8e86742093fe6e77 (DEFAULT_TABLE table=test-table [id=cdc66727646e43e0acb1dc97d470a735]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:53.312558 2477 tablet_service.cc:1505] Processing CreateTablet for tablet 7e63afde536d48de8e86742093fe6e77 (DEFAULT_TABLE table=test-table [id=cdc66727646e43e0acb1dc97d470a735]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:53.312558 2618 tablet_service.cc:1505] Processing CreateTablet for tablet 7e63afde536d48de8e86742093fe6e77 (DEFAULT_TABLE table=test-table [id=cdc66727646e43e0acb1dc97d470a735]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:53.313681 2336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e63afde536d48de8e86742093fe6e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.313838 2618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e63afde536d48de8e86742093fe6e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.313831 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7e63afde536d48de8e86742093fe6e77. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.323812 2707 tablet_bootstrap.cc:492] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Bootstrap starting.
I20251027 14:50:53.327033 2708 tablet_bootstrap.cc:492] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Bootstrap starting.
I20251027 14:50:53.327330 2707 tablet_bootstrap.cc:654] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.328580 2709 tablet_bootstrap.cc:492] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Bootstrap starting.
I20251027 14:50:53.328779 2707 log.cc:826] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:53.330400 2708 tablet_bootstrap.cc:654] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.331159 2709 tablet_bootstrap.cc:654] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.331684 2708 log.cc:826] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:53.332046 2709 log.cc:826] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Log is configured to *not* fsync() on all Append() calls
I20251027 14:50:53.334885 2708 tablet_bootstrap.cc:492] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: No bootstrap required, opened a new log
I20251027 14:50:53.334887 2707 tablet_bootstrap.cc:492] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: No bootstrap required, opened a new log
I20251027 14:50:53.334965 2709 tablet_bootstrap.cc:492] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: No bootstrap required, opened a new log
I20251027 14:50:53.335175 2707 ts_tablet_manager.cc:1403] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Time spent bootstrapping tablet: real 0.012s user 0.003s sys 0.006s
I20251027 14:50:53.335165 2708 ts_tablet_manager.cc:1403] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20251027 14:50:53.335203 2709 ts_tablet_manager.cc:1403] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251027 14:50:53.339952 2709 raft_consensus.cc:359] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.340317 2709 raft_consensus.cc:385] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.340436 2709 raft_consensus.cc:740] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.341167 2709 consensus_queue.cc:260] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.342165 2547 heartbeater.cc:499] Master 127.0.104.62:46743 was elected leader, sending a full tablet report...
I20251027 14:50:53.342008 2707 raft_consensus.cc:359] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.342381 2707 raft_consensus.cc:385] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.342481 2707 raft_consensus.cc:740] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02454102e084805bdbebeab42c0538e, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.342770 2709 ts_tablet_manager.cc:1434] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent starting tablet: real 0.007s user 0.005s sys 0.003s
I20251027 14:50:53.343071 2707 consensus_queue.cc:260] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.343631 2708 raft_consensus.cc:359] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.343971 2708 raft_consensus.cc:385] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.344050 2708 raft_consensus.cc:740] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.344638 2707 ts_tablet_manager.cc:1434] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Time spent starting tablet: real 0.009s user 0.009s sys 0.002s
I20251027 14:50:53.344751 2708 consensus_queue.cc:260] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.345736 2688 heartbeater.cc:499] Master 127.0.104.62:46743 was elected leader, sending a full tablet report...
I20251027 14:50:53.346101 2708 ts_tablet_manager.cc:1434] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Time spent starting tablet: real 0.011s user 0.010s sys 0.000s
W20251027 14:50:53.425031 2407 tablet.cc:2378] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:50:53.462570 2548 tablet.cc:2378] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:50:53.467206 2689 tablet.cc:2378] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:50:53.502017 2715 raft_consensus.cc:493] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:53.502296 2715 raft_consensus.cc:515] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.503986 2715 leader_election.cc:290] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959), b02454102e084805bdbebeab42c0538e (127.0.104.1:37543)
I20251027 14:50:53.508972 2577 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.504312 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.104.3:49645)
1027 14:50:53.504506 (+ 194us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.504525 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.504665 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.505221 (+ 556us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.505231 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.505264 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.505693 (+ 429us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.505706 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.507090 (+ 1384us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.507096 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.508483 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.508500 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.508609 (+ 109us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.508628 (+ 19us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.508674 (+ 46us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.508762 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":78}
I20251027 14:50:53.509850 2716 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.504714 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37543 (local address 127.0.104.3:38109)
1027 14:50:53.505364 (+ 650us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.505381 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.505490 (+ 109us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.505777 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.505781 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.505805 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.506414 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.506427 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.507797 (+ 1370us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.507801 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.509153 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.509167 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.509385 (+ 218us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.509410 (+ 25us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.509479 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.509589 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":420,"threads_started":1}
I20251027 14:50:53.510128 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.504328 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:49645 (local address 127.0.104.2:36959)
1027 14:50:53.504497 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.504501 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.504593 (+ 92us) server_negotiation.cc:408] Connection header received
1027 14:50:53.504817 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.504824 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.504903 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.505010 (+ 107us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.505833 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.506959 (+ 1126us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.508632 (+ 1673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.509674 (+ 1042us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.509723 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.509795 (+ 72us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.509916 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:50:53.510514 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.504929 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:38109 (local address 127.0.104.1:37543)
1027 14:50:53.505139 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.505146 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.505467 (+ 321us) server_negotiation.cc:408] Connection header received
1027 14:50:53.505600 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.505606 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.505672 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.505766 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.506550 (+ 784us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.507603 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.509347 (+ 1744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.510116 (+ 769us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.510163 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.510244 (+ 81us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.510354 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251027 14:50:53.510964 2502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e63afde536d48de8e86742093fe6e77" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" is_pre_election: true
I20251027 14:50:53.511487 2502 raft_consensus.cc:2468] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 0.
I20251027 14:50:53.511446 2360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e63afde536d48de8e86742093fe6e77" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e" is_pre_election: true
I20251027 14:50:53.511883 2360 raft_consensus.cc:2468] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 0.
I20251027 14:50:53.512111 2571 leader_election.cc:304] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3261285c7144e3aa5bd41920599a35, 4b59f5c96e6d453999a2713bde3dd5b2; no voters:
I20251027 14:50:53.512473 2715 raft_consensus.cc:2804] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:53.512599 2715 raft_consensus.cc:493] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:53.512697 2715 raft_consensus.cc:3060] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:53.514842 2715 raft_consensus.cc:515] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.515385 2715 leader_election.cc:290] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 election: Requested vote from peers 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959), b02454102e084805bdbebeab42c0538e (127.0.104.1:37543)
I20251027 14:50:53.515748 2502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e63afde536d48de8e86742093fe6e77" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b59f5c96e6d453999a2713bde3dd5b2"
I20251027 14:50:53.515811 2360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "7e63afde536d48de8e86742093fe6e77" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e"
I20251027 14:50:53.515913 2502 raft_consensus.cc:3060] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:53.515971 2360 raft_consensus.cc:3060] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:53.518218 2360 raft_consensus.cc:2468] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 1.
I20251027 14:50:53.518227 2502 raft_consensus.cc:2468] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 1.
I20251027 14:50:53.518585 2571 leader_election.cc:304] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3261285c7144e3aa5bd41920599a35, b02454102e084805bdbebeab42c0538e; no voters:
I20251027 14:50:53.518844 2715 raft_consensus.cc:2804] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:53.519237 2715 raft_consensus.cc:697] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Running, Role: LEADER
I20251027 14:50:53.519742 2715 consensus_queue.cc:237] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } }
I20251027 14:50:53.524647 2195 catalog_manager.cc:5649] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0d3261285c7144e3aa5bd41920599a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } health_report { overall_health: UNKNOWN } } }
Oct 27 14:50:53 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576651, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:53.574512 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.563157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46743 (local address 127.0.0.1:53756)
1027 14:50:53.563699 (+ 542us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.563716 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.563855 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.564122 (+ 267us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.564127 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.564879 (+ 752us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:50:53.565302 (+ 423us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.565319 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.566521 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.566525 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.567269 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.567279 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.567458 (+ 179us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.568119 (+ 661us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:50:53.568140 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:50:53.570153 (+ 2013us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:50:53.572657 (+ 2504us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:53.572667 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:53.572674 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:53.573018 (+ 344us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:53.573270 (+ 252us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:50:53.573273 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:50:53.573276 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:50:53.573360 (+ 84us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:50:53.573743 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:50:53.573749 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:50:53.573888 (+ 139us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.574002 (+ 114us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.574221 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":431,"thread_start_us":315,"threads_started":1}
I20251027 14:50:53.574559 2294 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.563164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53756 (local address 127.0.104.62:46743)
1027 14:50:53.563339 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.563343 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.563803 (+ 460us) server_negotiation.cc:408] Connection header received
1027 14:50:53.563960 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.563964 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.564014 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.564090 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:53.565453 (+ 1363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.566361 (+ 908us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.567395 (+ 1034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.567847 (+ 452us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.570287 (+ 2440us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:53.570314 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:53.570318 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:53.570344 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:53.572552 (+ 2208us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.573106 (+ 554us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.573110 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.573112 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.573179 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:53.573446 (+ 267us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:53.573449 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:53.573451 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:53.573648 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:53.573749 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.574072 (+ 323us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.574324 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":72,"server-negotiator.queue_time_us":55}
I20251027 14:50:53.579036 2195 catalog_manager.cc:2257] Servicing CreateTable request from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:53756:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251027 14:50:53.579955 2195 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:53.586438 2477 tablet_service.cc:1505] Processing CreateTablet for tablet 6bfe777d20db431095ee4bd74bed0310 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd6edd2f91184d0e89041acee9bec028]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:50:53.586228 2336 tablet_service.cc:1505] Processing CreateTablet for tablet 6bfe777d20db431095ee4bd74bed0310 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd6edd2f91184d0e89041acee9bec028]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:50:53.587206 2336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bfe777d20db431095ee4bd74bed0310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.587205 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bfe777d20db431095ee4bd74bed0310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.586761 2618 tablet_service.cc:1505] Processing CreateTablet for tablet 6bfe777d20db431095ee4bd74bed0310 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=bd6edd2f91184d0e89041acee9bec028]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:50:53.587509 2618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6bfe777d20db431095ee4bd74bed0310. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.593585 2708 tablet_bootstrap.cc:492] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: Bootstrap starting.
I20251027 14:50:53.594672 2709 tablet_bootstrap.cc:492] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: Bootstrap starting.
I20251027 14:50:53.595189 2708 tablet_bootstrap.cc:654] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.595366 2707 tablet_bootstrap.cc:492] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: Bootstrap starting.
I20251027 14:50:53.596303 2709 tablet_bootstrap.cc:654] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.596956 2707 tablet_bootstrap.cc:654] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.597535 2708 tablet_bootstrap.cc:492] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: No bootstrap required, opened a new log
I20251027 14:50:53.597638 2708 ts_tablet_manager.cc:1403] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:50:53.598459 2709 tablet_bootstrap.cc:492] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: No bootstrap required, opened a new log
I20251027 14:50:53.598380 2708 raft_consensus.cc:359] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.598606 2708 raft_consensus.cc:385] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.598609 2709 ts_tablet_manager.cc:1403] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:50:53.598697 2708 raft_consensus.cc:740] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.598974 2708 consensus_queue.cc:260] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.599433 2709 raft_consensus.cc:359] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.599551 2708 ts_tablet_manager.cc:1434] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 14:50:53.599609 2709 raft_consensus.cc:385] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.599491 2715 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.599689 2709 raft_consensus.cc:740] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.599748 2715 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.599900 2709 consensus_queue.cc:260] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.600358 2709 ts_tablet_manager.cc:1434] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 14:50:53.600279 2713 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.600625 2713 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.602679 2707 tablet_bootstrap.cc:492] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: No bootstrap required, opened a new log
I20251027 14:50:53.602833 2707 ts_tablet_manager.cc:1403] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20251027 14:50:53.603474 2707 raft_consensus.cc:359] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.603618 2707 raft_consensus.cc:385] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.603704 2707 raft_consensus.cc:740] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02454102e084805bdbebeab42c0538e, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.603919 2707 consensus_queue.cc:260] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.604363 2707 ts_tablet_manager.cc:1434] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: Time spent starting tablet: real 0.001s user 0.000s sys 0.000s
I20251027 14:50:53.604329 2714 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.604712 2714 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.701483 2713 raft_consensus.cc:493] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:53.701701 2713 raft_consensus.cc:515] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.703714 2713 leader_election.cc:290] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b02454102e084805bdbebeab42c0538e (127.0.104.1:37543), 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:53.709529 2728 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.704087 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37543 (local address 127.0.104.2:42133)
1027 14:50:53.704714 (+ 627us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.704732 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.704869 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.705413 (+ 544us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.705417 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.705438 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.706055 (+ 617us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.706064 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.707381 (+ 1317us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.707387 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.708783 (+ 1396us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.708798 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.708980 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.709007 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.709077 (+ 70us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.709197 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":518,"thread_start_us":388,"threads_started":1}
I20251027 14:50:53.710333 2705 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.704178 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:42133 (local address 127.0.104.1:37543)
1027 14:50:53.704314 (+ 136us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.704318 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.704807 (+ 489us) server_negotiation.cc:408] Connection header received
1027 14:50:53.705061 (+ 254us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.705070 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.705167 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.705229 (+ 62us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.706193 (+ 964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.707187 (+ 994us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.708929 (+ 1742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.709953 (+ 1024us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.709994 (+ 41us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.710061 (+ 67us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.710159 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251027 14:50:53.711316 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.705028 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:45349 (local address 127.0.104.3:43281)
1027 14:50:53.705193 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.705199 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.705441 (+ 242us) server_negotiation.cc:408] Connection header received
1027 14:50:53.705603 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.705610 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.705666 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.705737 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:53.706600 (+ 863us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.707522 (+ 922us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.709494 (+ 1972us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.710527 (+ 1033us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.710572 (+ 45us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.710968 (+ 396us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.711101 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251027 14:50:53.711337 2729 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.704866 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:43281 (local address 127.0.104.2:45349)
1027 14:50:53.705342 (+ 476us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.705355 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.705487 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.705768 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.705777 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.705802 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:53.706464 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.706477 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.707716 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.707721 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.709324 (+ 1603us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.709338 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.710781 (+ 1443us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.710804 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.710903 (+ 99us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.711028 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":263,"threads_started":1}
I20251027 14:50:53.712577 2360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bfe777d20db431095ee4bd74bed0310" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e" is_pre_election: true
I20251027 14:50:53.712778 2360 raft_consensus.cc:2468] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 0.
I20251027 14:50:53.713230 2430 leader_election.cc:304] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b59f5c96e6d453999a2713bde3dd5b2, b02454102e084805bdbebeab42c0538e; no voters:
I20251027 14:50:53.713610 2643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bfe777d20db431095ee4bd74bed0310" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3261285c7144e3aa5bd41920599a35" is_pre_election: true
I20251027 14:50:53.713816 2713 raft_consensus.cc:2804] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:53.713917 2713 raft_consensus.cc:493] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:53.713917 2643 raft_consensus.cc:2468] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 0.
I20251027 14:50:53.713984 2713 raft_consensus.cc:3060] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:53.716463 2713 raft_consensus.cc:515] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.717116 2713 leader_election.cc:290] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 election: Requested vote from peers b02454102e084805bdbebeab42c0538e (127.0.104.1:37543), 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:53.717502 2360 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bfe777d20db431095ee4bd74bed0310" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e"
I20251027 14:50:53.717604 2643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "6bfe777d20db431095ee4bd74bed0310" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3261285c7144e3aa5bd41920599a35"
I20251027 14:50:53.717682 2360 raft_consensus.cc:3060] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:53.717770 2643 raft_consensus.cc:3060] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:53.719777 2360 raft_consensus.cc:2468] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 1.
I20251027 14:50:53.720166 2430 leader_election.cc:304] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b59f5c96e6d453999a2713bde3dd5b2, b02454102e084805bdbebeab42c0538e; no voters:
I20251027 14:50:53.720467 2713 raft_consensus.cc:2804] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:53.720916 2713 raft_consensus.cc:697] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Running, Role: LEADER
I20251027 14:50:53.720958 2643 raft_consensus.cc:2468] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 1.
I20251027 14:50:53.721690 2713 consensus_queue.cc:237] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.725322 2730 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: TxnStatusTablet state changed. Reason: New leader 4b59f5c96e6d453999a2713bde3dd5b2. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.725637 2730 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:50:53.726289 2732 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251027 14:50:53.726559 2732 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251027 14:50:53.727079 2197 catalog_manager.cc:5649] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } health_report { overall_health: UNKNOWN } } }
I20251027 14:50:53.755015 2724 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.749429 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.0.1:40680)
1027 14:50:53.749608 (+ 179us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.749632 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.749772 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.750207 (+ 435us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.750211 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.750221 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:53.750539 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.750546 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.752215 (+ 1669us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.752219 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.753061 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.753073 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.753199 (+ 126us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.753244 (+ 45us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.754649 (+ 1405us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.754660 (+ 11us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.754705 (+ 45us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.754803 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":42}
I20251027 14:50:53.755115 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.749448 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40680 (local address 127.0.104.2:36959)
1027 14:50:53.749623 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.749630 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.749700 (+ 70us) server_negotiation.cc:408] Connection header received
1027 14:50:53.749941 (+ 241us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.749948 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.750052 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.750208 (+ 156us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:53.750658 (+ 450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.752090 (+ 1432us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.753206 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.753682 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.753767 (+ 85us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.754542 (+ 775us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.754700 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.754746 (+ 46us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.754840 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251027 14:50:53.763195 2360 raft_consensus.cc:1275] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Refusing update from remote peer 4b59f5c96e6d453999a2713bde3dd5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:53.763209 2643 raft_consensus.cc:1275] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Refusing update from remote peer 4b59f5c96e6d453999a2713bde3dd5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:53.764153 2730 consensus_queue.cc:1048] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:53.764452 2713 consensus_queue.cc:1048] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:53.774085 2715 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: TxnStatusTablet state changed. Reason: New leader 4b59f5c96e6d453999a2713bde3dd5b2. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.774338 2715 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.775420 2714 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: TxnStatusTablet state changed. Reason: New leader 4b59f5c96e6d453999a2713bde3dd5b2. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.775645 2714 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.779074 2713 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.779356 2713 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:50:53.784287 2713 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.784502 2713 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 4b59f5c96e6d453999a2713bde3dd5b2: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:50:53.802037 2715 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.802314 2715 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P 0d3261285c7144e3aa5bd41920599a35: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.805433 2714 tablet_replica.cc:442] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } } }
I20251027 14:50:53.805653 2714 tablet_replica.cc:445] T 6bfe777d20db431095ee4bd74bed0310 P b02454102e084805bdbebeab42c0538e: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:50:53.810215 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.802702 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40852 (local address 127.0.104.3:43281)
1027 14:50:53.802848 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.802854 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.803447 (+ 593us) server_negotiation.cc:408] Connection header received
1027 14:50:53.803758 (+ 311us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.803765 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.803869 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.803982 (+ 113us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:53.805678 (+ 1696us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.806957 (+ 1279us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.808268 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.808778 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.808865 (+ 87us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.809584 (+ 719us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.809725 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.809822 (+ 97us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.809914 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251027 14:50:53.825502 2748 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.820049 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.104.3:56497)
1027 14:50:53.820660 (+ 611us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.820674 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.820792 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.821144 (+ 352us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.821147 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.821156 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:53.821638 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.821646 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.823294 (+ 1648us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.823297 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.824072 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.824082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.824214 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.824256 (+ 42us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.825094 (+ 838us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.825114 (+ 20us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.825168 (+ 54us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.825256 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":469,"thread_start_us":339,"threads_started":1}
I20251027 14:50:53.825594 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.820024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:56497 (local address 127.0.104.2:36959)
1027 14:50:53.820169 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.820174 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.820764 (+ 590us) server_negotiation.cc:408] Connection header received
1027 14:50:53.820908 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.820914 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.821024 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.821122 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:53.821764 (+ 642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.823175 (+ 1411us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.824199 (+ 1024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.824767 (+ 568us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.824837 (+ 70us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.825001 (+ 164us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.825085 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.825210 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.825301 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251027 14:50:53.834612 2360 raft_consensus.cc:1275] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Refusing update from remote peer 0d3261285c7144e3aa5bd41920599a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:53.834868 2502 raft_consensus.cc:1275] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Refusing update from remote peer 0d3261285c7144e3aa5bd41920599a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:53.835090 2715 consensus_queue.cc:1048] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:53.835554 2717 consensus_queue.cc:1048] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:53.841740 2737 mvcc.cc:204] Tried to move back new op lower bound from 7215417974100963328 to 7215417972821237760. Current Snapshot: MvccSnapshot[applied={T|T < 7215417974100963328}]
I20251027 14:50:53.845247 2736 mvcc.cc:204] Tried to move back new op lower bound from 7215417974100963328 to 7215417972821237760. Current Snapshot: MvccSnapshot[applied={T|T < 7215417974100963328}]
I20251027 14:50:53.871482 2702 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.866556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:54743 (local address 127.0.104.3:43281)
1027 14:50:53.866719 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:53.866724 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:53.866946 (+ 222us) server_negotiation.cc:408] Connection header received
1027 14:50:53.867114 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:53.867120 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:53.867197 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:53.867270 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:53.868030 (+ 760us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.869037 (+ 1007us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.870368 (+ 1331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.870778 (+ 410us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.870830 (+ 52us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.870925 (+ 95us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.870999 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:53.871126 (+ 127us) server_negotiation.cc:300] Negotiation successful
1027 14:50:53.871208 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251027 14:50:53.871547 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:53.866208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:43281 (local address 127.0.104.2:54743)
1027 14:50:53.866848 (+ 640us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:53.866867 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:53.866999 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:53.867298 (+ 299us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:53.867303 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:53.867315 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:53.867907 (+ 592us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.867917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.869233 (+ 1316us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:53.869238 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:53.870254 (+ 1016us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:53.870265 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:53.870427 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:53.870491 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:53.871021 (+ 530us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:53.871036 (+ 15us) client_negotiation.cc:769] Sending connection context
1027 14:50:53.871103 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:50:53.871220 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":530,"threads_started":1}
I20251027 14:50:53.911110 2197 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:53.911362 2197 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:53.915315 2197 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Sending DeleteTablet for 3 replicas of tablet 7e63afde536d48de8e86742093fe6e77
I20251027 14:50:53.916736 2477 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e63afde536d48de8e86742093fe6e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:53 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40674
I20251027 14:50:53.916946 2336 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e63afde536d48de8e86742093fe6e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:53 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:32768
I20251027 14:50:53.917834 2618 tablet_service.cc:1552] Processing DeleteTablet for tablet 7e63afde536d48de8e86742093fe6e77 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:53 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40844
I20251027 14:50:53.925647 2197 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 14:50:53.926967 2756 tablet_replica.cc:333] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: stopping tablet replica
I20251027 14:50:53.927565 2756 raft_consensus.cc:2243] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:53.927956 2756 raft_consensus.cc:2272] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
W20251027 14:50:53.928666 2197 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:53.930207 2756 ts_tablet_manager.cc:1916] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:53.938561 2756 ts_tablet_manager.cc:1929] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:53.938818 2756 log.cc:1199] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/7e63afde536d48de8e86742093fe6e77
I20251027 14:50:53.939970 2335 tablet_service.cc:1505] Processing CreateTablet for tablet 297d0588ba10471e8f17e965b7af7d42 (DEFAULT_TABLE table=test-table [id=a460ef9d79f74312874e80af6c51f342]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:53.940532 2335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297d0588ba10471e8f17e965b7af7d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.940598 2477 tablet_service.cc:1505] Processing CreateTablet for tablet 297d0588ba10471e8f17e965b7af7d42 (DEFAULT_TABLE table=test-table [id=a460ef9d79f74312874e80af6c51f342]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:53.940941 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297d0588ba10471e8f17e965b7af7d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.945192 2617 tablet_service.cc:1505] Processing CreateTablet for tablet 297d0588ba10471e8f17e965b7af7d42 (DEFAULT_TABLE table=test-table [id=a460ef9d79f74312874e80af6c51f342]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:53.945657 2617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 297d0588ba10471e8f17e965b7af7d42. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:53.947283 2707 tablet_bootstrap.cc:492] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Bootstrap starting.
I20251027 14:50:53.948958 2707 tablet_bootstrap.cc:654] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.957932 2709 tablet_bootstrap.cc:492] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Bootstrap starting.
I20251027 14:50:53.959769 2709 tablet_bootstrap.cc:654] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.960287 2758 tablet_replica.cc:333] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: stopping tablet replica
I20251027 14:50:53.961898 2758 raft_consensus.cc:2243] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:50:53.962582 2758 raft_consensus.cc:2272] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:53.963380 2757 tablet_replica.cc:333] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: stopping tablet replica
I20251027 14:50:53.963563 2708 tablet_bootstrap.cc:492] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Bootstrap starting.
I20251027 14:50:53.963788 2757 raft_consensus.cc:2243] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:53.964138 2757 raft_consensus.cc:2272] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:53.965435 2708 tablet_bootstrap.cc:654] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:53.966156 2758 ts_tablet_manager.cc:1916] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:53.967675 2709 tablet_bootstrap.cc:492] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: No bootstrap required, opened a new log
I20251027 14:50:53.967839 2709 ts_tablet_manager.cc:1403] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent bootstrapping tablet: real 0.010s user 0.001s sys 0.004s
I20251027 14:50:53.968683 2708 tablet_bootstrap.cc:492] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: No bootstrap required, opened a new log
I20251027 14:50:53.968601 2709 raft_consensus.cc:359] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.968791 2709 raft_consensus.cc:385] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.968842 2708 ts_tablet_manager.cc:1403] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Time spent bootstrapping tablet: real 0.005s user 0.000s sys 0.004s
I20251027 14:50:53.968855 2709 raft_consensus.cc:740] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.969045 2756 ts_tablet_manager.cc:1950] T 7e63afde536d48de8e86742093fe6e77 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting consensus metadata
I20251027 14:50:53.969090 2709 consensus_queue.cc:260] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.969568 2709 ts_tablet_manager.cc:1434] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 14:50:53.969568 2708 raft_consensus.cc:359] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.969735 2708 raft_consensus.cc:385] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.969800 2708 raft_consensus.cc:740] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.970013 2708 consensus_queue.cc:260] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.970497 2708 ts_tablet_manager.cc:1434] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20251027 14:50:53.971930 2757 ts_tablet_manager.cc:1916] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:53.976964 2183 catalog_manager.cc:4985] TS 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959): tablet 7e63afde536d48de8e86742093fe6e77 (table test-table [id=cdc66727646e43e0acb1dc97d470a735]) successfully deleted
I20251027 14:50:53.981101 2758 ts_tablet_manager.cc:1929] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:53.981254 2758 log.cc:1199] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/7e63afde536d48de8e86742093fe6e77
I20251027 14:50:53.981580 2707 tablet_bootstrap.cc:492] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: No bootstrap required, opened a new log
I20251027 14:50:53.981704 2758 ts_tablet_manager.cc:1950] T 7e63afde536d48de8e86742093fe6e77 P 0d3261285c7144e3aa5bd41920599a35: Deleting consensus metadata
I20251027 14:50:53.981755 2707 ts_tablet_manager.cc:1403] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Time spent bootstrapping tablet: real 0.035s user 0.008s sys 0.006s
I20251027 14:50:53.982539 2707 raft_consensus.cc:359] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.982738 2707 raft_consensus.cc:385] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:53.982815 2707 raft_consensus.cc:740] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02454102e084805bdbebeab42c0538e, State: Initialized, Role: FOLLOWER
I20251027 14:50:53.983044 2707 consensus_queue.cc:260] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:53.983412 2183 catalog_manager.cc:4985] TS 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281): tablet 7e63afde536d48de8e86742093fe6e77 (table test-table [id=cdc66727646e43e0acb1dc97d470a735]) successfully deleted
I20251027 14:50:53.983515 2707 ts_tablet_manager.cc:1434] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 14:50:53.987578 2757 ts_tablet_manager.cc:1929] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:53.987752 2757 log.cc:1199] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/7e63afde536d48de8e86742093fe6e77
I20251027 14:50:53.988214 2757 ts_tablet_manager.cc:1950] T 7e63afde536d48de8e86742093fe6e77 P b02454102e084805bdbebeab42c0538e: Deleting consensus metadata
I20251027 14:50:53.990124 2183 catalog_manager.cc:4985] TS b02454102e084805bdbebeab42c0538e (127.0.104.1:37543): tablet 7e63afde536d48de8e86742093fe6e77 (table test-table [id=cdc66727646e43e0acb1dc97d470a735]) successfully deleted
I20251027 14:50:54.036494 2741 raft_consensus.cc:493] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:54.036718 2741 raft_consensus.cc:515] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:54.037549 2741 leader_election.cc:290] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b02454102e084805bdbebeab42c0538e (127.0.104.1:37543), 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:54.038208 2643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "297d0588ba10471e8f17e965b7af7d42" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3261285c7144e3aa5bd41920599a35" is_pre_election: true
I20251027 14:50:54.038297 2359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "297d0588ba10471e8f17e965b7af7d42" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e" is_pre_election: true
I20251027 14:50:54.038432 2643 raft_consensus.cc:2468] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 0.
I20251027 14:50:54.038523 2359 raft_consensus.cc:2468] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 0.
I20251027 14:50:54.039062 2430 leader_election.cc:304] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b59f5c96e6d453999a2713bde3dd5b2, b02454102e084805bdbebeab42c0538e; no voters:
I20251027 14:50:54.039381 2741 raft_consensus.cc:2804] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:54.039502 2741 raft_consensus.cc:493] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:54.039592 2741 raft_consensus.cc:3060] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:54.043031 2741 raft_consensus.cc:515] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:54.043668 2741 leader_election.cc:290] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 election: Requested vote from peers b02454102e084805bdbebeab42c0538e (127.0.104.1:37543), 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:54.044203 2643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "297d0588ba10471e8f17e965b7af7d42" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3261285c7144e3aa5bd41920599a35"
I20251027 14:50:54.044260 2359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "297d0588ba10471e8f17e965b7af7d42" candidate_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e"
I20251027 14:50:54.044421 2643 raft_consensus.cc:3060] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:54.044468 2359 raft_consensus.cc:3060] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:54.047591 2643 raft_consensus.cc:2468] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 1.
I20251027 14:50:54.048012 2359 raft_consensus.cc:2468] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4b59f5c96e6d453999a2713bde3dd5b2 in term 1.
I20251027 14:50:54.048039 2430 leader_election.cc:304] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3261285c7144e3aa5bd41920599a35, 4b59f5c96e6d453999a2713bde3dd5b2; no voters:
I20251027 14:50:54.048569 2741 raft_consensus.cc:2804] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:54.048746 2741 raft_consensus.cc:697] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 LEADER]: Becoming Leader. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Running, Role: LEADER
I20251027 14:50:54.049077 2741 consensus_queue.cc:237] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:54.053417 2197 catalog_manager.cc:5649] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 reported cstate change: term changed from 0 to 1, leader changed from <none> to 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } health_report { overall_health: UNKNOWN } } }
I20251027 14:50:54.088428 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:54.082841 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40688 (local address 127.0.104.2:36959)
1027 14:50:54.083002 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:54.083008 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:54.083038 (+ 30us) server_negotiation.cc:408] Connection header received
1027 14:50:54.083205 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:54.083211 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:54.083296 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:54.083400 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:54.084048 (+ 648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:54.085441 (+ 1393us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:54.087018 (+ 1577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:54.087523 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:54.087594 (+ 71us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:54.087757 (+ 163us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:54.087849 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:54.087993 (+ 144us) server_negotiation.cc:300] Negotiation successful
1027 14:50:54.088099 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:50:54.104488 2752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:54.098229 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.104.2:36711)
1027 14:50:54.098415 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:54.098436 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:54.098548 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:54.099146 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:54.099151 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:54.099163 (+ 12us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:54.099616 (+ 453us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:54.099630 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:54.101449 (+ 1819us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:54.101455 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:54.102712 (+ 1257us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:54.102727 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:54.102843 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:54.102882 (+ 39us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:54.103915 (+ 1033us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:54.103926 (+ 11us) client_negotiation.cc:769] Sending connection context
1027 14:50:54.103982 (+ 56us) client_negotiation.cc:240] Negotiation successful
1027 14:50:54.104144 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251027 14:50:54.105162 2704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:54.098724 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:36711 (local address 127.0.104.2:36959)
1027 14:50:54.098859 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:54.098865 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:54.098880 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:50:54.098937 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:54.098943 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:54.099016 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:54.099753 (+ 737us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:54.099932 (+ 179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:54.101314 (+ 1382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:54.103053 (+ 1739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:54.103568 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:54.103651 (+ 83us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:54.103795 (+ 144us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:54.104497 (+ 702us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:54.104545 (+ 48us) server_negotiation.cc:300] Negotiation successful
1027 14:50:54.104872 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251027 14:50:54.116137 2359 raft_consensus.cc:1275] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Refusing update from remote peer 4b59f5c96e6d453999a2713bde3dd5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:54.116173 2643 raft_consensus.cc:1275] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Refusing update from remote peer 4b59f5c96e6d453999a2713bde3dd5b2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:54.116778 2730 consensus_queue.cc:1048] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:54.117033 2713 consensus_queue.cc:1048] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 14:50:54.154806 2735 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251027 14:50:55.771682 2197 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:55.771888 2197 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53750:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:55.775467 2197 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Sending DeleteTablet for 3 replicas of tablet 297d0588ba10471e8f17e965b7af7d42
I20251027 14:50:55.776217 2477 tablet_service.cc:1552] Processing DeleteTablet for tablet 297d0588ba10471e8f17e965b7af7d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:55 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40674
I20251027 14:50:55.776222 2336 tablet_service.cc:1552] Processing DeleteTablet for tablet 297d0588ba10471e8f17e965b7af7d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:55 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:32768
I20251027 14:50:55.776295 2617 tablet_service.cc:1552] Processing DeleteTablet for tablet 297d0588ba10471e8f17e965b7af7d42 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:55 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40844
I20251027 14:50:55.788165 2789 tablet_replica.cc:333] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: stopping tablet replica
I20251027 14:50:55.788522 2789 raft_consensus.cc:2243] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:50:55.789084 2791 tablet_replica.cc:333] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: stopping tablet replica
I20251027 14:50:55.789227 2789 raft_consensus.cc:2272] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:55.789413 2791 raft_consensus.cc:2243] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:55.789675 2791 raft_consensus.cc:2272] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:55.790211 2790 tablet_replica.cc:333] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: stopping tablet replica
I20251027 14:50:55.790362 2789 ts_tablet_manager.cc:1916] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:55.790390 2790 raft_consensus.cc:2243] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:55.790575 2790 raft_consensus.cc:2272] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:55.791054 2791 ts_tablet_manager.cc:1916] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:55.791507 2790 ts_tablet_manager.cc:1916] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Deleting tablet data with delete state TABLET_DATA_DELETED
Oct 27 14:50:55 dist-test-slave-hxxt krb5kdc[2149](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
I20251027 14:50:55.815541 2789 ts_tablet_manager.cc:1929] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251027 14:50:55.815672 2789 log.cc:1199] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/297d0588ba10471e8f17e965b7af7d42
I20251027 14:50:55.816066 2789 ts_tablet_manager.cc:1950] T 297d0588ba10471e8f17e965b7af7d42 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting consensus metadata
I20251027 14:50:55.816768 2791 ts_tablet_manager.cc:1929] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251027 14:50:55.816887 2791 log.cc:1199] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/297d0588ba10471e8f17e965b7af7d42
I20251027 14:50:55.817257 2791 ts_tablet_manager.cc:1950] T 297d0588ba10471e8f17e965b7af7d42 P 0d3261285c7144e3aa5bd41920599a35: Deleting consensus metadata
I20251027 14:50:55.874608 2183 catalog_manager.cc:4985] TS 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959): tablet 297d0588ba10471e8f17e965b7af7d42 (table test-table [id=a460ef9d79f74312874e80af6c51f342]) successfully deleted
W20251027 14:50:55.907471 416 mini_kdc.cc:317] Time spent kinit for test-user: real 0.130s user 0.003s sys 0.009s
I20251027 14:50:55.907621 2790 ts_tablet_manager.cc:1929] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251027 14:50:55.907781 2790 log.cc:1199] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/297d0588ba10471e8f17e965b7af7d42
I20251027 14:50:55.908219 2790 ts_tablet_manager.cc:1950] T 297d0588ba10471e8f17e965b7af7d42 P b02454102e084805bdbebeab42c0538e: Deleting consensus metadata
I20251027 14:50:55.908478 2183 catalog_manager.cc:4985] TS 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281): tablet 297d0588ba10471e8f17e965b7af7d42 (table test-table [id=a460ef9d79f74312874e80af6c51f342]) successfully deleted
I20251027 14:50:55.909410 2183 catalog_manager.cc:4985] TS b02454102e084805bdbebeab42c0538e (127.0.104.1:37543): tablet 297d0588ba10471e8f17e965b7af7d42 (table test-table [id=a460ef9d79f74312874e80af6c51f342]) successfully deleted
Oct 27 14:50:55 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:50:55.923579 2799 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:55.910794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53760 (local address 127.0.104.62:46743)
1027 14:50:55.911312 (+ 518us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:55.911316 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:55.911335 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:50:55.911484 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:55.911488 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:55.911552 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:55.911642 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:55.913107 (+ 1465us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:55.914096 (+ 989us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:55.915288 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:55.915601 (+ 313us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:55.919081 (+ 3480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:55.919123 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:55.919127 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:55.919158 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:55.921519 (+ 2361us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:55.922106 (+ 587us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:55.922111 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:55.922113 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:55.922201 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:55.922486 (+ 285us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:55.922489 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:55.922491 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:55.922713 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:55.922897 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:55.923155 (+ 258us) server_negotiation.cc:300] Negotiation successful
1027 14:50:55.923321 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":422,"thread_start_us":268,"threads_started":1}
I20251027 14:50:55.927227 2197 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:50:55.927771 2197 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:55.933909 2336 tablet_service.cc:1505] Processing CreateTablet for tablet bc3aceac95134befa75bfdd4ef872e00 (DEFAULT_TABLE table=test-table [id=7e8362b977ab4177b2f39146b0247ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:55.934350 2336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3aceac95134befa75bfdd4ef872e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:55.934317 2477 tablet_service.cc:1505] Processing CreateTablet for tablet bc3aceac95134befa75bfdd4ef872e00 (DEFAULT_TABLE table=test-table [id=7e8362b977ab4177b2f39146b0247ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:55.934628 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3aceac95134befa75bfdd4ef872e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:55.939337 2800 tablet_bootstrap.cc:492] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Bootstrap starting.
I20251027 14:50:55.939328 2617 tablet_service.cc:1505] Processing CreateTablet for tablet bc3aceac95134befa75bfdd4ef872e00 (DEFAULT_TABLE table=test-table [id=7e8362b977ab4177b2f39146b0247ecc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:55.939736 2617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bc3aceac95134befa75bfdd4ef872e00. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:55.940994 2800 tablet_bootstrap.cc:654] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:55.941387 2801 tablet_bootstrap.cc:492] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Bootstrap starting.
I20251027 14:50:55.943077 2801 tablet_bootstrap.cc:654] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:55.943135 2800 tablet_bootstrap.cc:492] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: No bootstrap required, opened a new log
I20251027 14:50:55.943259 2800 ts_tablet_manager.cc:1403] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:50:55.944072 2800 raft_consensus.cc:359] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:55.944325 2800 raft_consensus.cc:385] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:55.944459 2800 raft_consensus.cc:740] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02454102e084805bdbebeab42c0538e, State: Initialized, Role: FOLLOWER
I20251027 14:50:55.944700 2800 consensus_queue.cc:260] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:55.946192 2801 tablet_bootstrap.cc:492] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: No bootstrap required, opened a new log
I20251027 14:50:55.946321 2801 ts_tablet_manager.cc:1403] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251027 14:50:55.946897 2801 raft_consensus.cc:359] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:55.947093 2801 raft_consensus.cc:385] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:55.947181 2801 raft_consensus.cc:740] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Initialized, Role: FOLLOWER
I20251027 14:50:55.947413 2801 consensus_queue.cc:260] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:55.949887 2801 ts_tablet_manager.cc:1434] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 14:50:55.951400 2805 tablet_bootstrap.cc:492] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Bootstrap starting.
I20251027 14:50:55.952899 2800 ts_tablet_manager.cc:1434] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Time spent starting tablet: real 0.010s user 0.007s sys 0.003s
I20251027 14:50:55.953375 2805 tablet_bootstrap.cc:654] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:55.955318 2805 tablet_bootstrap.cc:492] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: No bootstrap required, opened a new log
I20251027 14:50:55.955446 2805 ts_tablet_manager.cc:1403] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Time spent bootstrapping tablet: real 0.004s user 0.000s sys 0.003s
I20251027 14:50:55.956036 2805 raft_consensus.cc:359] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:55.956198 2805 raft_consensus.cc:385] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:55.956285 2805 raft_consensus.cc:740] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Initialized, Role: FOLLOWER
I20251027 14:50:55.956508 2805 consensus_queue.cc:260] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:55.957290 2805 ts_tablet_manager.cc:1434] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 14:50:56.021063 2804 raft_consensus.cc:493] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:56.021262 2804 raft_consensus.cc:515] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.023433 2804 leader_election.cc:290] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959), 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:56.029438 2809 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.023543 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.104.1:37243)
1027 14:50:56.023988 (+ 445us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:56.024005 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:56.024150 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:56.024594 (+ 444us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:56.024603 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:56.024634 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:56.025313 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.025331 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.027085 (+ 1754us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.027090 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:56.028709 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.028723 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.028893 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.028923 (+ 30us) client_negotiation.cc:769] Sending connection context
1027 14:50:56.029012 (+ 89us) client_negotiation.cc:240] Negotiation successful
1027 14:50:56.029130 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":227,"threads_started":1}
I20251027 14:50:56.029711 2810 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.024166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:43281 (local address 127.0.104.1:54503)
1027 14:50:56.024697 (+ 531us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:56.024720 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:56.024840 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:56.025261 (+ 421us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:56.025264 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:56.025282 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:50:56.025793 (+ 511us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.025803 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.027351 (+ 1548us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.027356 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:56.029011 (+ 1655us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.029023 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.029165 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.029194 (+ 29us) client_negotiation.cc:769] Sending connection context
1027 14:50:56.029314 (+ 120us) client_negotiation.cc:240] Negotiation successful
1027 14:50:56.029422 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":281,"threads_started":1}
I20251027 14:50:56.030731 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.023127 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:37243 (local address 127.0.104.2:36959)
1027 14:50:56.023785 (+ 658us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:56.023793 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:56.024099 (+ 306us) server_negotiation.cc:408] Connection header received
1027 14:50:56.024298 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:56.024306 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:56.024421 (+ 115us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:56.025398 (+ 977us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:56.025605 (+ 207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.026930 (+ 1325us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.028873 (+ 1943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.030154 (+ 1281us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.030206 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:56.030310 (+ 104us) server_negotiation.cc:300] Negotiation successful
1027 14:50:56.030436 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":353,"threads_started":1}
I20251027 14:50:56.031327 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.024311 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:54503 (local address 127.0.104.3:43281)
1027 14:50:56.024959 (+ 648us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:56.024966 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:56.024987 (+ 21us) server_negotiation.cc:408] Connection header received
1027 14:50:56.025053 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:56.025059 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:56.025136 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:56.025234 (+ 98us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:50:56.025938 (+ 704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.027173 (+ 1235us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.029155 (+ 1982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.030787 (+ 1632us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.030834 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:56.030936 (+ 102us) server_negotiation.cc:300] Negotiation successful
1027 14:50:56.031064 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":399,"threads_started":1}
I20251027 14:50:56.031322 2502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc3aceac95134befa75bfdd4ef872e00" candidate_uuid: "b02454102e084805bdbebeab42c0538e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" is_pre_election: true
I20251027 14:50:56.031535 2502 raft_consensus.cc:2468] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b02454102e084805bdbebeab42c0538e in term 0.
I20251027 14:50:56.031874 2643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc3aceac95134befa75bfdd4ef872e00" candidate_uuid: "b02454102e084805bdbebeab42c0538e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3261285c7144e3aa5bd41920599a35" is_pre_election: true
I20251027 14:50:56.032094 2643 raft_consensus.cc:2468] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate b02454102e084805bdbebeab42c0538e in term 0.
I20251027 14:50:56.032058 2288 leader_election.cc:304] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b59f5c96e6d453999a2713bde3dd5b2, b02454102e084805bdbebeab42c0538e; no voters:
I20251027 14:50:56.032501 2804 raft_consensus.cc:2804] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:56.032614 2804 raft_consensus.cc:493] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:56.032708 2804 raft_consensus.cc:3060] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:56.035020 2804 raft_consensus.cc:515] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.035552 2804 leader_election.cc:290] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [CANDIDATE]: Term 1 election: Requested vote from peers 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959), 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281)
I20251027 14:50:56.035951 2502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc3aceac95134befa75bfdd4ef872e00" candidate_uuid: "b02454102e084805bdbebeab42c0538e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b59f5c96e6d453999a2713bde3dd5b2"
I20251027 14:50:56.036010 2643 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bc3aceac95134befa75bfdd4ef872e00" candidate_uuid: "b02454102e084805bdbebeab42c0538e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3261285c7144e3aa5bd41920599a35"
I20251027 14:50:56.036116 2502 raft_consensus.cc:3060] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:56.036190 2643 raft_consensus.cc:3060] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:56.038236 2502 raft_consensus.cc:2468] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b02454102e084805bdbebeab42c0538e in term 1.
I20251027 14:50:56.038365 2643 raft_consensus.cc:2468] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate b02454102e084805bdbebeab42c0538e in term 1.
I20251027 14:50:56.038569 2288 leader_election.cc:304] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4b59f5c96e6d453999a2713bde3dd5b2, b02454102e084805bdbebeab42c0538e; no voters:
I20251027 14:50:56.038839 2804 raft_consensus.cc:2804] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:56.039211 2804 raft_consensus.cc:697] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 1 LEADER]: Becoming Leader. State: Replica: b02454102e084805bdbebeab42c0538e, State: Running, Role: LEADER
I20251027 14:50:56.039569 2804 consensus_queue.cc:237] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.042770 2197 catalog_manager.cc:5649] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e reported cstate change: term changed from 0 to 1, leader changed from <none> to b02454102e084805bdbebeab42c0538e (127.0.104.1). New cstate: current_term: 1 leader_uuid: "b02454102e084805bdbebeab42c0538e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } health_report { overall_health: UNKNOWN } } }
I20251027 14:50:56.076093 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.070893 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32778 (local address 127.0.104.1:37543)
1027 14:50:56.071391 (+ 498us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:56.071395 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:56.071414 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:50:56.071490 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:56.071494 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:56.071567 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:56.071691 (+ 124us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:56.072215 (+ 524us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.073280 (+ 1065us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.074540 (+ 1260us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.074866 (+ 326us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.074935 (+ 69us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:56.075518 (+ 583us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:56.075624 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:56.075749 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:50:56.075821 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":321,"threads_started":1}
I20251027 14:50:56.091053 2808 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.086496 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:50653 (local address 127.0.104.2:36959)
1027 14:50:56.086630 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:56.086635 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:56.086990 (+ 355us) server_negotiation.cc:408] Connection header received
1027 14:50:56.087118 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:56.087123 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:56.087188 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:56.087264 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:56.088077 (+ 813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.089001 (+ 924us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.090061 (+ 1060us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.090359 (+ 298us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.090408 (+ 49us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:56.090517 (+ 109us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:56.090586 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:56.090753 (+ 167us) server_negotiation.cc:300] Negotiation successful
1027 14:50:56.090869 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:50:56.091068 2818 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.086301 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:36959 (local address 127.0.104.1:50653)
1027 14:50:56.086897 (+ 596us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:56.086913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:56.087023 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:56.087379 (+ 356us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:56.087384 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:56.087395 (+ 11us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:56.087914 (+ 519us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.087923 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.089155 (+ 1232us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.089160 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:56.089964 (+ 804us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.089973 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.090126 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.090173 (+ 47us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:56.090621 (+ 448us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:56.090632 (+ 11us) client_negotiation.cc:769] Sending connection context
1027 14:50:56.090705 (+ 73us) client_negotiation.cc:240] Negotiation successful
1027 14:50:56.090807 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":456,"thread_start_us":319,"threads_started":1}
I20251027 14:50:56.099429 2643 raft_consensus.cc:1275] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Refusing update from remote peer b02454102e084805bdbebeab42c0538e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:56.099541 2502 raft_consensus.cc:1275] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Refusing update from remote peer b02454102e084805bdbebeab42c0538e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:56.100015 2812 consensus_queue.cc:1048] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:56.100265 2804 consensus_queue.cc:1048] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:56.127664 2823 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.122798 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37543 (local address 127.0.104.2:38079)
1027 14:50:56.123314 (+ 516us) negotiation.cc:107] Waiting for socket to connect
1027 14:50:56.123329 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:50:56.123432 (+ 103us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:50:56.123705 (+ 273us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:50:56.123708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:50:56.123717 (+ 9us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:50:56.124211 (+ 494us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.124220 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.125562 (+ 1342us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.125566 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:50:56.126492 (+ 926us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:50:56.126504 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.126639 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.126668 (+ 29us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:50:56.127231 (+ 563us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:50:56.127241 (+ 10us) client_negotiation.cc:769] Sending connection context
1027 14:50:56.127312 (+ 71us) client_negotiation.cc:240] Negotiation successful
1027 14:50:56.127422 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":286,"threads_started":1}
I20251027 14:50:56.127691 2815 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.122856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:38079 (local address 127.0.104.1:37543)
1027 14:50:56.123009 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:56.123013 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:56.123396 (+ 383us) server_negotiation.cc:408] Connection header received
1027 14:50:56.123548 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:56.123552 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:56.123622 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:56.123690 (+ 68us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:56.124367 (+ 677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.125393 (+ 1026us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.126604 (+ 1211us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.126950 (+ 346us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.127006 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:56.127133 (+ 127us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:56.127202 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:56.127337 (+ 135us) server_negotiation.cc:300] Negotiation successful
1027 14:50:56.127412 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:50:56.159468 2197 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:56.159713 2197 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:56.162662 2197 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Sending DeleteTablet for 3 replicas of tablet bc3aceac95134befa75bfdd4ef872e00
I20251027 14:50:56.163467 2336 tablet_service.cc:1552] Processing DeleteTablet for tablet bc3aceac95134befa75bfdd4ef872e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:56 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:32768
I20251027 14:50:56.163573 2617 tablet_service.cc:1552] Processing DeleteTablet for tablet bc3aceac95134befa75bfdd4ef872e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:56 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40844
I20251027 14:50:56.163736 2790 tablet_replica.cc:333] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: stopping tablet replica
I20251027 14:50:56.163827 2791 tablet_replica.cc:333] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: stopping tablet replica
I20251027 14:50:56.163918 2790 raft_consensus.cc:2243] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:50:56.164017 2791 raft_consensus.cc:2243] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:56.164202 2791 raft_consensus.cc:2272] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:56.164646 2790 raft_consensus.cc:2272] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:56.164548 2197 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53760:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:50:56.165103 2197 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:50:56.165351 2791 ts_tablet_manager.cc:1916] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:56.165905 2790 ts_tablet_manager.cc:1916] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:56.168409 2477 tablet_service.cc:1552] Processing DeleteTablet for tablet bc3aceac95134befa75bfdd4ef872e00 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:56 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40674
I20251027 14:50:56.168674 2789 tablet_replica.cc:333] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: stopping tablet replica
I20251027 14:50:56.168861 2789 raft_consensus.cc:2243] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:56.169063 2789 raft_consensus.cc:2272] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:56.170151 2789 ts_tablet_manager.cc:1916] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:56.173024 2790 ts_tablet_manager.cc:1929] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:56.173170 2790 log.cc:1199] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/bc3aceac95134befa75bfdd4ef872e00
I20251027 14:50:56.173589 2790 ts_tablet_manager.cc:1950] T bc3aceac95134befa75bfdd4ef872e00 P b02454102e084805bdbebeab42c0538e: Deleting consensus metadata
I20251027 14:50:56.174953 2183 catalog_manager.cc:4985] TS b02454102e084805bdbebeab42c0538e (127.0.104.1:37543): tablet bc3aceac95134befa75bfdd4ef872e00 (table test-table [id=7e8362b977ab4177b2f39146b0247ecc]) successfully deleted
I20251027 14:50:56.176987 2789 ts_tablet_manager.cc:1929] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:56.177208 2789 log.cc:1199] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/bc3aceac95134befa75bfdd4ef872e00
I20251027 14:50:56.177661 2789 ts_tablet_manager.cc:1950] T bc3aceac95134befa75bfdd4ef872e00 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting consensus metadata
I20251027 14:50:56.178448 2336 tablet_service.cc:1505] Processing CreateTablet for tablet 37eb9bfbd819445ea17ccfe82f1501a4 (DEFAULT_TABLE table=test-table [id=9d066855dca5429586b55fb3f42977d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:56.178860 2336 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37eb9bfbd819445ea17ccfe82f1501a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:56.178882 2791 ts_tablet_manager.cc:1929] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:50:56.179075 2791 log.cc:1199] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/bc3aceac95134befa75bfdd4ef872e00
I20251027 14:50:56.179487 2791 ts_tablet_manager.cc:1950] T bc3aceac95134befa75bfdd4ef872e00 P 0d3261285c7144e3aa5bd41920599a35: Deleting consensus metadata
I20251027 14:50:56.180743 2183 catalog_manager.cc:4985] TS 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281): tablet bc3aceac95134befa75bfdd4ef872e00 (table test-table [id=7e8362b977ab4177b2f39146b0247ecc]) successfully deleted
I20251027 14:50:56.180989 2183 catalog_manager.cc:4985] TS 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959): tablet bc3aceac95134befa75bfdd4ef872e00 (table test-table [id=7e8362b977ab4177b2f39146b0247ecc]) successfully deleted
I20251027 14:50:56.181262 2617 tablet_service.cc:1505] Processing CreateTablet for tablet 37eb9bfbd819445ea17ccfe82f1501a4 (DEFAULT_TABLE table=test-table [id=9d066855dca5429586b55fb3f42977d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:56.181612 2617 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37eb9bfbd819445ea17ccfe82f1501a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:56.187592 2800 tablet_bootstrap.cc:492] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Bootstrap starting.
I20251027 14:50:56.187873 2477 tablet_service.cc:1505] Processing CreateTablet for tablet 37eb9bfbd819445ea17ccfe82f1501a4 (DEFAULT_TABLE table=test-table [id=9d066855dca5429586b55fb3f42977d4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:50:56.188274 2477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37eb9bfbd819445ea17ccfe82f1501a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:50:56.189407 2800 tablet_bootstrap.cc:654] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:56.194989 2801 tablet_bootstrap.cc:492] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Bootstrap starting.
I20251027 14:50:56.195340 2805 tablet_bootstrap.cc:492] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Bootstrap starting.
I20251027 14:50:56.196712 2801 tablet_bootstrap.cc:654] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:56.197058 2805 tablet_bootstrap.cc:654] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Neither blocks nor log segments found. Creating new log.
I20251027 14:50:56.206707 2805 tablet_bootstrap.cc:492] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: No bootstrap required, opened a new log
I20251027 14:50:56.207016 2805 ts_tablet_manager.cc:1403] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.005s
I20251027 14:50:56.207257 2801 tablet_bootstrap.cc:492] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: No bootstrap required, opened a new log
I20251027 14:50:56.207435 2801 ts_tablet_manager.cc:1403] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.007s
I20251027 14:50:56.208030 2805 raft_consensus.cc:359] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.208150 2801 raft_consensus.cc:359] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.208302 2800 tablet_bootstrap.cc:492] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: No bootstrap required, opened a new log
I20251027 14:50:56.208351 2801 raft_consensus.cc:385] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:56.208428 2805 raft_consensus.cc:385] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:56.208508 2801 raft_consensus.cc:740] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b59f5c96e6d453999a2713bde3dd5b2, State: Initialized, Role: FOLLOWER
I20251027 14:50:56.208564 2805 raft_consensus.cc:740] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Initialized, Role: FOLLOWER
I20251027 14:50:56.208587 2800 ts_tablet_manager.cc:1403] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.013s
I20251027 14:50:56.208787 2801 consensus_queue.cc:260] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.208981 2805 consensus_queue.cc:260] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.209421 2800 raft_consensus.cc:359] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.209622 2800 raft_consensus.cc:385] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:50:56.209692 2800 raft_consensus.cc:740] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b02454102e084805bdbebeab42c0538e, State: Initialized, Role: FOLLOWER
I20251027 14:50:56.209966 2800 consensus_queue.cc:260] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.210471 2800 ts_tablet_manager.cc:1434] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 14:50:56.210757 2801 ts_tablet_manager.cc:1434] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 14:50:56.213680 2805 ts_tablet_manager.cc:1434] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20251027 14:50:56.344280 2807 raft_consensus.cc:493] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:50:56.344508 2807 raft_consensus.cc:515] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.345338 2807 leader_election.cc:290] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b02454102e084805bdbebeab42c0538e (127.0.104.1:37543), 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959)
I20251027 14:50:56.345814 2359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37eb9bfbd819445ea17ccfe82f1501a4" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e" is_pre_election: true
I20251027 14:50:56.345844 2502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37eb9bfbd819445ea17ccfe82f1501a4" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" is_pre_election: true
I20251027 14:50:56.346045 2359 raft_consensus.cc:2468] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 0.
I20251027 14:50:56.346071 2502 raft_consensus.cc:2468] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 0.
I20251027 14:50:56.346470 2571 leader_election.cc:304] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3261285c7144e3aa5bd41920599a35, 4b59f5c96e6d453999a2713bde3dd5b2; no voters:
I20251027 14:50:56.346766 2807 raft_consensus.cc:2804] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:50:56.346885 2807 raft_consensus.cc:493] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:50:56.346997 2807 raft_consensus.cc:3060] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:56.349104 2807 raft_consensus.cc:515] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.349596 2807 leader_election.cc:290] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 election: Requested vote from peers b02454102e084805bdbebeab42c0538e (127.0.104.1:37543), 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959)
I20251027 14:50:56.350106 2502 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37eb9bfbd819445ea17ccfe82f1501a4" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4b59f5c96e6d453999a2713bde3dd5b2"
I20251027 14:50:56.350318 2502 raft_consensus.cc:3060] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:56.350735 2359 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "37eb9bfbd819445ea17ccfe82f1501a4" candidate_uuid: "0d3261285c7144e3aa5bd41920599a35" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b02454102e084805bdbebeab42c0538e"
I20251027 14:50:56.350937 2359 raft_consensus.cc:3060] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:50:56.353384 2502 raft_consensus.cc:2468] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 1.
I20251027 14:50:56.353847 2571 leader_election.cc:304] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3261285c7144e3aa5bd41920599a35, 4b59f5c96e6d453999a2713bde3dd5b2; no voters:
I20251027 14:50:56.354068 2359 raft_consensus.cc:2468] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3261285c7144e3aa5bd41920599a35 in term 1.
I20251027 14:50:56.354137 2807 raft_consensus.cc:2804] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:50:56.354571 2807 raft_consensus.cc:697] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 1 LEADER]: Becoming Leader. State: Replica: 0d3261285c7144e3aa5bd41920599a35, State: Running, Role: LEADER
I20251027 14:50:56.354897 2807 consensus_queue.cc:237] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } }
I20251027 14:50:56.357658 2197 catalog_manager.cc:5649] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 reported cstate change: term changed from 0 to 1, leader changed from <none> to 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "0d3261285c7144e3aa5bd41920599a35" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "0d3261285c7144e3aa5bd41920599a35" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43281 } health_report { overall_health: HEALTHY } } }
I20251027 14:50:56.397063 2811 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:56.392684 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40856 (local address 127.0.104.3:43281)
1027 14:50:56.392834 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:56.392839 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:56.392925 (+ 86us) server_negotiation.cc:408] Connection header received
1027 14:50:56.393121 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:56.393127 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:56.393206 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:56.393291 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:50:56.393868 (+ 577us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.394736 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:56.395945 (+ 1209us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:56.396312 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:56.396370 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:50:56.396510 (+ 140us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:50:56.396585 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:56.396731 (+ 146us) server_negotiation.cc:300] Negotiation successful
1027 14:50:56.396821 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:50:56.409606 2502 raft_consensus.cc:1275] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Refusing update from remote peer 0d3261285c7144e3aa5bd41920599a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:56.410326 2829 consensus_queue.cc:1048] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4b59f5c96e6d453999a2713bde3dd5b2" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 36959 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:50:56.410889 2359 raft_consensus.cc:1275] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Refusing update from remote peer 0d3261285c7144e3aa5bd41920599a35: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:50:56.411453 2829 consensus_queue.cc:1048] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b02454102e084805bdbebeab42c0538e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37543 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 14:50:56.437723 2832 tablet_replica.cc:1306] Aborted: operation has been aborted: cancelling pending write operations
I20251027 14:50:58.017905 2195 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:58.018095 2195 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:53760:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:50:58.021572 2195 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 4d74a3c2ce28416b880846cb7275ab04: Sending DeleteTablet for 3 replicas of tablet 37eb9bfbd819445ea17ccfe82f1501a4
I20251027 14:50:58.022385 2336 tablet_service.cc:1552] Processing DeleteTablet for tablet 37eb9bfbd819445ea17ccfe82f1501a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:58 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:32768
I20251027 14:50:58.022428 2617 tablet_service.cc:1552] Processing DeleteTablet for tablet 37eb9bfbd819445ea17ccfe82f1501a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:58 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40844
I20251027 14:50:58.022430 2476 tablet_service.cc:1552] Processing DeleteTablet for tablet 37eb9bfbd819445ea17ccfe82f1501a4 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:50:58 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40674
I20251027 14:50:58.023032 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2270
I20251027 14:50:58.037655 2858 tablet_replica.cc:333] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: stopping tablet replica
I20251027 14:50:58.038018 2859 tablet_replica.cc:333] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: stopping tablet replica
I20251027 14:50:58.037997 2858 raft_consensus.cc:2243] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:58.038229 2859 raft_consensus.cc:2243] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:50:58.038321 2858 raft_consensus.cc:2272] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:58.038632 2859 raft_consensus.cc:2272] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:58.039609 2859 ts_tablet_manager.cc:1916] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:58.039911 2858 ts_tablet_manager.cc:1916] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:58.046778 2859 ts_tablet_manager.cc:1929] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251027 14:50:58.046977 2859 log.cc:1199] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/37eb9bfbd819445ea17ccfe82f1501a4
I20251027 14:50:58.046967 2858 ts_tablet_manager.cc:1929] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251027 14:50:58.047277 2858 log.cc:1199] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/37eb9bfbd819445ea17ccfe82f1501a4
I20251027 14:50:58.047469 2859 ts_tablet_manager.cc:1950] T 37eb9bfbd819445ea17ccfe82f1501a4 P 0d3261285c7144e3aa5bd41920599a35: Deleting consensus metadata
I20251027 14:50:58.047683 2858 ts_tablet_manager.cc:1950] T 37eb9bfbd819445ea17ccfe82f1501a4 P 4b59f5c96e6d453999a2713bde3dd5b2: Deleting consensus metadata
I20251027 14:50:58.048647 2183 catalog_manager.cc:4985] TS 4b59f5c96e6d453999a2713bde3dd5b2 (127.0.104.2:36959): tablet 37eb9bfbd819445ea17ccfe82f1501a4 (table test-table [id=9d066855dca5429586b55fb3f42977d4]) successfully deleted
I20251027 14:50:58.048889 2183 catalog_manager.cc:4985] TS 0d3261285c7144e3aa5bd41920599a35 (127.0.104.3:43281): tablet 37eb9bfbd819445ea17ccfe82f1501a4 (table test-table [id=9d066855dca5429586b55fb3f42977d4]) successfully deleted
I20251027 14:50:58.049404 2857 tablet_replica.cc:333] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: stopping tablet replica
I20251027 14:50:58.049563 2857 raft_consensus.cc:2243] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:50:58.049750 2857 raft_consensus.cc:2272] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:50:58.050516 2857 ts_tablet_manager.cc:1916] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:50:58.055291 2857 ts_tablet_manager.cc:1929] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.8
I20251027 14:50:58.055419 2857 log.cc:1199] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TxnSmokeWithDifferentUserTypes.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/37eb9bfbd819445ea17ccfe82f1501a4
I20251027 14:50:58.055843 2857 ts_tablet_manager.cc:1950] T 37eb9bfbd819445ea17ccfe82f1501a4 P b02454102e084805bdbebeab42c0538e: Deleting consensus metadata
Oct 27 14:50:58 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:50:58.056730 2183 catalog_manager.cc:4985] TS b02454102e084805bdbebeab42c0538e (127.0.104.1:37543): tablet 37eb9bfbd819445ea17ccfe82f1501a4 (table test-table [id=9d066855dca5429586b55fb3f42977d4]) successfully deleted
I20251027 14:50:58.061991 2861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:58.026368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:32784 (local address 127.0.104.1:37543)
1027 14:50:58.047869 (+ 21501us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:58.047876 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:58.047901 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:50:58.047973 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:58.047980 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:58.048065 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:58.048178 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:58.050240 (+ 2062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:58.051253 (+ 1013us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:58.052427 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:58.052864 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:58.056859 (+ 3995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:58.056913 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:58.056930 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:58.056960 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:58.059607 (+ 2647us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:58.060229 (+ 622us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:58.060236 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:58.060239 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:58.060309 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:58.060618 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:58.060622 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:58.060624 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:58.060902 (+ 278us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:58.061175 (+ 273us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:58.061441 (+ 266us) server_negotiation.cc:300] Negotiation successful
1027 14:50:58.061678 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":21368,"thread_start_us":22705,"threads_started":1}
W20251027 14:50:58.062592 2401 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:32784
W20251027 14:50:58.063220 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2270: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:50:58.063395 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 2270
I20251027 14:50:58.081180 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2412
Oct 27 14:50:58 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:50:58.094416 2862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:58.081844 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40690 (local address 127.0.104.2:36959)
1027 14:50:58.082294 (+ 450us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:58.082299 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:58.082316 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:50:58.082366 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:58.082370 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:58.082428 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:58.082524 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:58.083851 (+ 1327us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:58.084940 (+ 1089us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:58.086051 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:58.086348 (+ 297us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:58.089524 (+ 3176us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:58.089571 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:58.089586 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:58.089615 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:58.092501 (+ 2886us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:58.093009 (+ 508us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:58.093016 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:58.093018 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:58.093086 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:58.093293 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:58.093296 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:58.093298 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:58.093533 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:58.093753 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:58.093922 (+ 169us) server_negotiation.cc:300] Negotiation successful
1027 14:50:58.094087 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":232,"threads_started":1}
W20251027 14:50:58.095046 2542 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40690
W20251027 14:50:58.095559 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2412: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:50:58.095697 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 2412
I20251027 14:50:58.115818 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2553
Oct 27 14:50:58 dist-test-slave-hxxt krb5kdc[2149](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576655, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:50:58.129874 2863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:50:58.116565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40870 (local address 127.0.104.3:43281)
1027 14:50:58.117148 (+ 583us) server_negotiation.cc:207] Beginning negotiation
1027 14:50:58.117153 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:50:58.117169 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:50:58.117221 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:50:58.117226 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:50:58.117282 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:50:58.117375 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:50:58.118724 (+ 1349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:58.119657 (+ 933us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:50:58.120764 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:50:58.121077 (+ 313us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:50:58.124353 (+ 3276us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:50:58.124432 (+ 79us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:50:58.124449 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:50:58.124480 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:50:58.127290 (+ 2810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:58.127930 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:58.127938 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:58.127940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:58.128022 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:50:58.128310 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:50:58.128315 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:50:58.128318 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:50:58.128735 (+ 417us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:50:58.129026 (+ 291us) server_negotiation.cc:1036] Waiting for connection context
1027 14:50:58.129359 (+ 333us) server_negotiation.cc:300] Negotiation successful
1027 14:50:58.129584 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":491,"thread_start_us":401,"threads_started":1}
W20251027 14:50:58.130563 2683 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:40870
W20251027 14:50:58.131191 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2553: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:50:58.131333 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 2553
I20251027 14:50:58.149415 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2165
I20251027 14:50:58.150509 2237 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:50:58.264230 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 2165
2025-10-27T14:50:58Z chronyd exiting
[ OK ] SecurityITest.TxnSmokeWithDifferentUserTypes (9280 ms)
[ RUN ] SecurityITest.TestNoKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:50:58 dist-test-slave-hxxt krb5kdc[2867](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:50:58 dist-test-slave-hxxt krb5kdc[2867](info): set up 2 sockets
Oct 27 14:50:58 dist-test-slave-hxxt krb5kdc[2867](info): commencing operation
krb5kdc: starting...
W20251027 14:51:00.128491 416 mini_kdc.cc:121] Time spent starting KDC: real 1.825s user 0.006s sys 0.010s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:00 dist-test-slave-hxxt krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:00Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:00.341531 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35129
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35129
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:00.650833 2883 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:00.651127 2883 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:00.651249 2883 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:00.659054 2883 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:00.659140 2883 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:00.659224 2883 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:00.659286 2883 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:00.659363 2883 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:00.669405 2883 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35129
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35129
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:00.670905 2883 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:00.672184 2883 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:00.680073 2889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:00.680042 2888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:00.680680 2883 server_base.cc:1047] running on GCE node
W20251027 14:51:00.680814 2891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:00.681612 2883 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:00.682932 2883 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:00.684142 2883 hybrid_clock.cc:648] HybridClock initialized: now 1761576660684076 us; error 44 us; skew 500 ppm
I20251027 14:51:00.684458 2883 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:00 dist-test-slave-hxxt krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576660, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:00.687448 2883 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:00.688935 2883 webserver.cc:492] Webserver started at http://127.0.104.62:42221/ using document root <none> and password file <none>
I20251027 14:51:00.689620 2883 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:00.689697 2883 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:00.689996 2883 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:00.692524 2883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "5eaad357173545938bbc5bb5d284a657"
format_stamp: "Formatted at 2025-10-27 14:51:00 on dist-test-slave-hxxt"
server_key: "0a819e587dc69c3d4faf859edc56a83d"
server_key_iv: "2d6a01bb5a149eb8f10800f02265a9b1"
server_key_version: "encryptionkey@0"
I20251027 14:51:00.693233 2883 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "5eaad357173545938bbc5bb5d284a657"
format_stamp: "Formatted at 2025-10-27 14:51:00 on dist-test-slave-hxxt"
server_key: "0a819e587dc69c3d4faf859edc56a83d"
server_key_iv: "2d6a01bb5a149eb8f10800f02265a9b1"
server_key_version: "encryptionkey@0"
I20251027 14:51:00.698372 2883 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:51:00.701512 2898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:00.702751 2883 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:00.702906 2883 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "5eaad357173545938bbc5bb5d284a657"
format_stamp: "Formatted at 2025-10-27 14:51:00 on dist-test-slave-hxxt"
server_key: "0a819e587dc69c3d4faf859edc56a83d"
server_key_iv: "2d6a01bb5a149eb8f10800f02265a9b1"
server_key_version: "encryptionkey@0"
I20251027 14:51:00.703094 2883 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:00.724583 2883 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:00.728464 2883 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:00.728806 2883 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:00.747124 2883 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35129
I20251027 14:51:00.747164 2949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35129 every 8 connection(s)
I20251027 14:51:00.748570 2883 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:00.752559 2950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:00.757259 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 2883
I20251027 14:51:00.757442 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:00.758036 416 external_mini_cluster.cc:1468] Setting key 20abb47257ecb6176585afb4f67c8217
I20251027 14:51:00.764273 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657: Bootstrap starting.
I20251027 14:51:00.767948 2950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:00.769263 2950 log.cc:826] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657: Log is configured to *not* fsync() on all Append() calls
Oct 27 14:51:00 dist-test-slave-hxxt krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576660, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:00.772713 2950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657: No bootstrap required, opened a new log
I20251027 14:51:00.778630 2950 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eaad357173545938bbc5bb5d284a657" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35129 } }
I20251027 14:51:00.778968 2950 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:00.779068 2950 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5eaad357173545938bbc5bb5d284a657, State: Initialized, Role: FOLLOWER
I20251027 14:51:00.779255 2953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:00.760007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33720 (local address 127.0.104.62:35129)
1027 14:51:00.760848 (+ 841us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:00.760863 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:00.760919 (+ 56us) server_negotiation.cc:408] Connection header received
1027 14:51:00.762530 (+ 1611us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:00.762559 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:00.763114 (+ 555us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:00.763602 (+ 488us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:00.765059 (+ 1457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:00.767070 (+ 2011us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:00.768213 (+ 1143us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:00.768823 (+ 610us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:00.772548 (+ 3725us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:00.772607 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:00.772622 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:00.772655 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:00.775476 (+ 2821us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:00.776226 (+ 750us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:00.776232 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:00.776238 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:00.776344 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:00.776650 (+ 306us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:00.776653 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:00.776655 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:00.777130 (+ 475us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:00.777385 (+ 255us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:00.777777 (+ 392us) server_negotiation.cc:300] Negotiation successful
1027 14:51:00.778155 (+ 378us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":395,"threads_started":1}
I20251027 14:51:00.779691 2950 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eaad357173545938bbc5bb5d284a657" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35129 } }
I20251027 14:51:00.779871 2950 raft_consensus.cc:399] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:00.780001 2950 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:00.780170 2950 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:00.782444 2950 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eaad357173545938bbc5bb5d284a657" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35129 } }
I20251027 14:51:00.782919 2950 leader_election.cc:304] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 5eaad357173545938bbc5bb5d284a657; no voters:
I20251027 14:51:00.783444 2950 leader_election.cc:290] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:00.783604 2955 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:00.784135 2955 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [term 1 LEADER]: Becoming Leader. State: Replica: 5eaad357173545938bbc5bb5d284a657, State: Running, Role: LEADER
I20251027 14:51:00.784639 2955 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eaad357173545938bbc5bb5d284a657" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35129 } }
I20251027 14:51:00.785290 2950 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:00.787226 2957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5eaad357173545938bbc5bb5d284a657. Latest consensus state: current_term: 1 leader_uuid: "5eaad357173545938bbc5bb5d284a657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eaad357173545938bbc5bb5d284a657" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35129 } } }
I20251027 14:51:00.787513 2957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:00.789846 2962 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:00.789752 2956 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "5eaad357173545938bbc5bb5d284a657" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "5eaad357173545938bbc5bb5d284a657" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35129 } } }
I20251027 14:51:00.790007 2956 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:00.794580 2962 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:00.803905 2962 catalog_manager.cc:1357] Generated new cluster ID: 9a8b822ed5e94492b2d4bce3084e5a02
I20251027 14:51:00.804028 2962 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:00.856058 2962 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:00.857275 2962 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:00.882030 2962 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 5eaad357173545938bbc5bb5d284a657: Generated new TSK 0
I20251027 14:51:00.882956 2962 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
W20251027 14:51:01.885201 416 mini_kdc.cc:279] Time spent creating service keytab for HTTP/127.0.104.1: real 0.958s user 0.000s sys 0.008s
I20251027 14:51:01.886194 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35129
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:02.192628 2978 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:02.192940 2978 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:02.193089 2978 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:02.201157 2978 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:02.201260 2978 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:02.201409 2978 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:02.211732 2978 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:02.213253 2978 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:02.214529 2978 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:02.223268 2983 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:02.223273 2984 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:02.224439 2986 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:02.224435 2978 server_base.cc:1047] running on GCE node
I20251027 14:51:02.225031 2978 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:02.225898 2978 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:02.227101 2978 hybrid_clock.cc:648] HybridClock initialized: now 1761576662227038 us; error 40 us; skew 500 ppm
I20251027 14:51:02.227365 2978 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:02 dist-test-slave-hxxt krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:02.230417 2978 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:51:02.231900 2978 webserver.cc:492] Webserver started at http://127.0.104.1:33509/ using document root <none> and password file <none>
I20251027 14:51:02.232620 2978 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:02.232702 2978 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:02.233021 2978 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:02.235621 2978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "d15eb63f5c3f44768450626e3232ab98"
format_stamp: "Formatted at 2025-10-27 14:51:02 on dist-test-slave-hxxt"
server_key: "f6c412fa183f5ba656a5acff2cbad62e"
server_key_iv: "c04b2c1a21d1bf2cc96612701092271c"
server_key_version: "encryptionkey@0"
I20251027 14:51:02.236352 2978 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "d15eb63f5c3f44768450626e3232ab98"
format_stamp: "Formatted at 2025-10-27 14:51:02 on dist-test-slave-hxxt"
server_key: "f6c412fa183f5ba656a5acff2cbad62e"
server_key_iv: "c04b2c1a21d1bf2cc96612701092271c"
server_key_version: "encryptionkey@0"
I20251027 14:51:02.241688 2978 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20251027 14:51:02.244882 2993 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:02.246115 2978 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251027 14:51:02.246243 2978 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "d15eb63f5c3f44768450626e3232ab98"
format_stamp: "Formatted at 2025-10-27 14:51:02 on dist-test-slave-hxxt"
server_key: "f6c412fa183f5ba656a5acff2cbad62e"
server_key_iv: "c04b2c1a21d1bf2cc96612701092271c"
server_key_version: "encryptionkey@0"
I20251027 14:51:02.246416 2978 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:02.265722 2978 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:02.272902 2978 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:02.273135 2978 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:02.274051 2978 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:02.275767 2978 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:02.275851 2978 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:02.275925 2978 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:02.275975 2978 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:02.305502 2978 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37441
I20251027 14:51:02.305565 3105 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37441 every 8 connection(s)
I20251027 14:51:02.306789 2978 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:02.311888 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 2978
I20251027 14:51:02.312093 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:02.312705 416 external_mini_cluster.cc:1468] Setting key dcee38d03215718c7c8f86d50690fc04
Oct 27 14:51:02 dist-test-slave-hxxt krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:02.325055 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:02.309400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:52865 (local address 127.0.104.62:35129)
1027 14:51:02.309788 (+ 388us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:02.309793 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:02.310730 (+ 937us) server_negotiation.cc:408] Connection header received
1027 14:51:02.312234 (+ 1504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:02.312241 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:02.312336 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:02.312477 (+ 141us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:02.314477 (+ 2000us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.315750 (+ 1273us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:02.316897 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.317297 (+ 400us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:02.319967 (+ 2670us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:02.320007 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:02.320011 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:02.320041 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:02.322380 (+ 2339us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:02.323047 (+ 667us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:02.323052 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:02.323054 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:02.323121 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:02.323415 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:02.323419 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:02.323422 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:02.323625 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:02.323750 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:02.324464 (+ 714us) server_negotiation.cc:300] Negotiation successful
1027 14:51:02.324702 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":300,"thread_start_us":228,"threads_started":1}
I20251027 14:51:02.326032 3109 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:02.309901 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35129 (local address 127.0.104.1:52865)
1027 14:51:02.310570 (+ 669us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:02.310612 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:02.311926 (+ 1314us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:02.312739 (+ 813us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:02.312748 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:02.313377 (+ 629us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:02.314261 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:02.314285 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.315920 (+ 1635us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:02.315924 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:02.316771 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:02.316782 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.316988 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:02.317647 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:02.317671 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:02.319781 (+ 2110us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:02.322519 (+ 2738us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:02.322529 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:02.322547 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:02.322940 (+ 393us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:02.323211 (+ 271us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:02.323216 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:02.323218 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:02.323319 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:02.323736 (+ 417us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:02.323744 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:02.324044 (+ 300us) client_negotiation.cc:769] Sending connection context
1027 14:51:02.324336 (+ 292us) client_negotiation.cc:240] Negotiation successful
1027 14:51:02.324859 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":281,"threads_started":1}
I20251027 14:51:02.327596 3106 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35129
I20251027 14:51:02.327998 3106 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:02.329005 3106 heartbeater.cc:507] Master 127.0.104.62:35129 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:51:02.331249 2915 ts_manager.cc:194] Registered new tserver with Master: d15eb63f5c3f44768450626e3232ab98 (127.0.104.1:37441)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:51:02.332862 2915 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:52865
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:02.380357 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35129
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:02.697944 3114 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:02.698228 3114 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:02.698341 3114 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:02.706279 3114 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:02.706379 3114 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:02.706522 3114 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:02.716822 3114 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:02.718333 3114 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:02.719588 3114 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:02.728538 3120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:02.728469 3119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:02.728818 3122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:02.729768 3114 server_base.cc:1047] running on GCE node
I20251027 14:51:02.730324 3114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:02.731216 3114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:02.732451 3114 hybrid_clock.cc:648] HybridClock initialized: now 1761576662732398 us; error 49 us; skew 500 ppm
I20251027 14:51:02.732728 3114 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:02 dist-test-slave-hxxt krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:02.736157 3114 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:51:02.737951 3114 webserver.cc:492] Webserver started at http://127.0.104.2:34985/ using document root <none> and password file <none>
I20251027 14:51:02.738663 3114 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:02.738742 3114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:02.739058 3114 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:02.741575 3114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "8a7e398ab3144c9d8606b662d856e5cc"
format_stamp: "Formatted at 2025-10-27 14:51:02 on dist-test-slave-hxxt"
server_key: "2932a041727d93675673e8c3ea9bb128"
server_key_iv: "3fd9a77d2a816f54e1e63df763a1dcaf"
server_key_version: "encryptionkey@0"
I20251027 14:51:02.742280 3114 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "8a7e398ab3144c9d8606b662d856e5cc"
format_stamp: "Formatted at 2025-10-27 14:51:02 on dist-test-slave-hxxt"
server_key: "2932a041727d93675673e8c3ea9bb128"
server_key_iv: "3fd9a77d2a816f54e1e63df763a1dcaf"
server_key_version: "encryptionkey@0"
I20251027 14:51:02.747483 3114 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:51:02.750792 3129 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:02.752084 3114 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:51:02.752224 3114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "8a7e398ab3144c9d8606b662d856e5cc"
format_stamp: "Formatted at 2025-10-27 14:51:02 on dist-test-slave-hxxt"
server_key: "2932a041727d93675673e8c3ea9bb128"
server_key_iv: "3fd9a77d2a816f54e1e63df763a1dcaf"
server_key_version: "encryptionkey@0"
I20251027 14:51:02.752452 3114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:02.767534 3114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:02.771071 3114 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:02.771309 3114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:02.772219 3114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:02.773708 3114 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:02.773809 3114 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:02.773917 3114 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:02.773991 3114 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:02.804056 3114 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34351
I20251027 14:51:02.804095 3241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34351 every 8 connection(s)
I20251027 14:51:02.805387 3114 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:02.806198 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3114
I20251027 14:51:02.806394 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:02.806845 416 external_mini_cluster.cc:1468] Setting key 03188a6b5857b94d7c59c2e9c0b19b02
Oct 27 14:51:02 dist-test-slave-hxxt krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576662, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:51:02.823554 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:02.808201 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:39913 (local address 127.0.104.62:35129)
1027 14:51:02.808479 (+ 278us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:02.808485 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:02.809829 (+ 1344us) server_negotiation.cc:408] Connection header received
1027 14:51:02.811284 (+ 1455us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:02.811291 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:02.811384 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:02.811473 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:02.813210 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.814040 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:02.815114 (+ 1074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.815560 (+ 446us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:02.818255 (+ 2695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:02.818290 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:02.818294 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:02.818323 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:02.820697 (+ 2374us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:02.821495 (+ 798us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:02.821499 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:02.821502 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:02.821575 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:02.821920 (+ 345us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:02.821923 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:02.821924 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:02.822114 (+ 190us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:02.822207 (+ 93us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:02.823160 (+ 953us) server_negotiation.cc:300] Negotiation successful
1027 14:51:02.823337 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":142}
I20251027 14:51:02.824700 3244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:02.808921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35129 (local address 127.0.104.2:39913)
1027 14:51:02.809679 (+ 758us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:02.809717 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:02.811017 (+ 1300us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:02.811721 (+ 704us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:02.811735 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:02.812268 (+ 533us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:02.813040 (+ 772us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:02.813059 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.814184 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:02.814189 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:02.814979 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:02.814990 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:02.815184 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:02.815812 (+ 628us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:02.815835 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:02.818068 (+ 2233us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:02.820825 (+ 2757us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:02.820834 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:02.820853 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:02.821388 (+ 535us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:02.821682 (+ 294us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:02.821687 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:02.821690 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:02.821832 (+ 142us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:02.822225 (+ 393us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:02.822233 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:02.822689 (+ 456us) client_negotiation.cc:769] Sending connection context
1027 14:51:02.823057 (+ 368us) client_negotiation.cc:240] Negotiation successful
1027 14:51:02.823491 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":363,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:51:02.826062 3242 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35129
I20251027 14:51:02.826463 3242 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:02.827382 3242 heartbeater.cc:507] Master 127.0.104.62:35129 requested a full tablet report, sending...
I20251027 14:51:02.828895 2915 ts_manager.cc:194] Registered new tserver with Master: 8a7e398ab3144c9d8606b662d856e5cc (127.0.104.2:34351)
I20251027 14:51:02.829749 2915 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:39913
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:02.876900 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35129
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36453
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:03.183838 3249 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:03.184134 3249 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:03.184226 3249 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:03.192122 3249 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:03.192227 3249 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:03.192390 3249 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:03.202589 3249 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36453
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35129
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:03.204079 3249 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:03.205348 3249 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:03.214414 3255 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:03.214551 3254 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:03.215049 3257 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:03.215693 3249 server_base.cc:1047] running on GCE node
I20251027 14:51:03.216275 3249 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:03.217196 3249 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:03.218412 3249 hybrid_clock.cc:648] HybridClock initialized: now 1761576663218349 us; error 36 us; skew 500 ppm
I20251027 14:51:03.218686 3249 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:03 dist-test-slave-hxxt krb5kdc[2867](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:03.221778 3249 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:51:03.223307 3249 webserver.cc:492] Webserver started at http://127.0.104.3:33679/ using document root <none> and password file <none>
I20251027 14:51:03.224025 3249 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:03.224117 3249 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:03.224541 3249 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:03.227061 3249 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "f571dc9aa9cc49ea93ca96d94c14390b"
format_stamp: "Formatted at 2025-10-27 14:51:03 on dist-test-slave-hxxt"
server_key: "bbe64cd571d6e42c9b1656d107fe077c"
server_key_iv: "6f87bcdd292deea706598d5b13877402"
server_key_version: "encryptionkey@0"
I20251027 14:51:03.227799 3249 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "f571dc9aa9cc49ea93ca96d94c14390b"
format_stamp: "Formatted at 2025-10-27 14:51:03 on dist-test-slave-hxxt"
server_key: "bbe64cd571d6e42c9b1656d107fe077c"
server_key_iv: "6f87bcdd292deea706598d5b13877402"
server_key_version: "encryptionkey@0"
I20251027 14:51:03.233172 3249 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20251027 14:51:03.236745 3264 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:03.238155 3249 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:03.238329 3249 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "f571dc9aa9cc49ea93ca96d94c14390b"
format_stamp: "Formatted at 2025-10-27 14:51:03 on dist-test-slave-hxxt"
server_key: "bbe64cd571d6e42c9b1656d107fe077c"
server_key_iv: "6f87bcdd292deea706598d5b13877402"
server_key_version: "encryptionkey@0"
I20251027 14:51:03.238543 3249 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:03.253899 3249 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:03.257835 3249 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:03.258082 3249 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:03.259035 3249 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:03.260556 3249 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:03.260653 3249 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:03.260754 3249 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:03.260816 3249 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:03.290498 3249 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:32783
I20251027 14:51:03.290573 3376 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:32783 every 8 connection(s)
I20251027 14:51:03.291869 3249 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:03.293025 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3249
I20251027 14:51:03.293234 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNoKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:03.293761 416 external_mini_cluster.cc:1468] Setting key 91cc66ff5bfcce06b13c7cfb2dd42d56
Oct 27 14:51:03 dist-test-slave-hxxt krb5kdc[2867](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576663, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:03.310402 3108 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:03.294829 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:47491 (local address 127.0.104.62:35129)
1027 14:51:03.295014 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:03.295020 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:03.296265 (+ 1245us) server_negotiation.cc:408] Connection header received
1027 14:51:03.297872 (+ 1607us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:03.297879 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:03.297958 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:03.298043 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:03.299618 (+ 1575us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:03.300535 (+ 917us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:03.301604 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:03.302025 (+ 421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:03.304721 (+ 2696us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:03.304752 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:03.304756 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:03.304785 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:03.307163 (+ 2378us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:03.308113 (+ 950us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:03.308119 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:03.308123 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:03.308191 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:03.308562 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:03.308576 (+ 14us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:03.308580 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:03.309144 (+ 564us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:03.309290 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:03.309985 (+ 695us) server_negotiation.cc:300] Negotiation successful
1027 14:51:03.310176 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:51:03.311129 3379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:03.295423 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35129 (local address 127.0.104.3:47491)
1027 14:51:03.296104 (+ 681us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:03.296167 (+ 63us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:03.297620 (+ 1453us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:03.298271 (+ 651us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:03.298280 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:03.298815 (+ 535us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:03.299453 (+ 638us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:03.299481 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:03.300684 (+ 1203us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:03.300689 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:03.301477 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:03.301488 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:03.301736 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:03.302453 (+ 717us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:03.302497 (+ 44us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:03.304522 (+ 2025us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:03.307337 (+ 2815us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:03.307346 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:03.307365 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:03.307922 (+ 557us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:03.308285 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:03.308289 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:03.308293 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:03.308444 (+ 151us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:03.309268 (+ 824us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:03.309274 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:03.309559 (+ 285us) client_negotiation.cc:769] Sending connection context
1027 14:51:03.309832 (+ 273us) client_negotiation.cc:240] Negotiation successful
1027 14:51:03.310173 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":457,"thread_start_us":310,"threads_started":1}
I20251027 14:51:03.312325 3377 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35129
I20251027 14:51:03.312644 3377 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:03.313349 3377 heartbeater.cc:507] Master 127.0.104.62:35129 requested a full tablet report, sending...
I20251027 14:51:03.314647 2915 ts_manager.cc:194] Registered new tserver with Master: f571dc9aa9cc49ea93ca96d94c14390b (127.0.104.3:32783)
I20251027 14:51:03.315430 2915 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:47491
I20251027 14:51:03.321034 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:51:03.335675 3106 heartbeater.cc:499] Master 127.0.104.62:35129 was elected leader, sending a full tablet report...
W20251027 14:51:03.347781 3108 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:03.345723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37892 (local address 127.0.104.62:35129)
1027 14:51:03.345914 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:03.345921 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:03.346244 (+ 323us) server_negotiation.cc:408] Connection header received
1027 14:51:03.346377 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:03.346381 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:03.346454 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:03.346537 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:03.347516 (+ 979us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:37892: BlockingRecv error: recv got EOF from 127.0.0.1:37892 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:51:03.350346 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2978
W20251027 14:51:03.352864 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2978: Not authorized: Client connection negotiation failed: client connection to 127.0.104.1:37441: server requires authentication, but client does not have Kerberos credentials available
I20251027 14:51:03.352998 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 2978
I20251027 14:51:03.365236 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3114
W20251027 14:51:03.367604 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3114: Not authorized: Client connection negotiation failed: client connection to 127.0.104.2:34351: server requires authentication, but client does not have Kerberos credentials available
I20251027 14:51:03.367733 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3114
I20251027 14:51:03.379652 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3249
W20251027 14:51:03.382117 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3249: Not authorized: Client connection negotiation failed: client connection to 127.0.104.3:32783: server requires authentication, but client does not have Kerberos credentials available
I20251027 14:51:03.382232 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3249
I20251027 14:51:03.394289 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 2883
I20251027 14:51:03.395411 2945 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:03.494935 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 2883
2025-10-27T14:51:03Z chronyd exiting
[ OK ] SecurityITest.TestNoKerberosCredentials (5228 ms)
[ RUN ] SecurityITest.TestRebalanceReportUnauthorized
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:03 dist-test-slave-hxxt krb5kdc[3394](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:03 dist-test-slave-hxxt krb5kdc[3394](info): set up 2 sockets
Oct 27 14:51:03 dist-test-slave-hxxt krb5kdc[3394](info): commencing operation
krb5kdc: starting...
W20251027 14:51:05.360658 416 mini_kdc.cc:121] Time spent starting KDC: real 1.828s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:05 dist-test-slave-hxxt krb5kdc[3394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:05.561443 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35789
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35789
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:05.873479 3410 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:05.873776 3410 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:05.873904 3410 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:05.881829 3410 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:05.881932 3410 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:05.882011 3410 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:05.882072 3410 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:05.882148 3410 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:05.892313 3410 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:05.893867 3410 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:05.895134 3410 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:05.902733 3415 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:05.902711 3416 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:05.903348 3418 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:05.904038 3410 server_base.cc:1047] running on GCE node
I20251027 14:51:05.904780 3410 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:05.906049 3410 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:05.907258 3410 hybrid_clock.cc:648] HybridClock initialized: now 1761576665907195 us; error 39 us; skew 500 ppm
I20251027 14:51:05.907546 3410 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:05 dist-test-slave-hxxt krb5kdc[3394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576665, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:05.910346 3410 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:05.911814 3410 webserver.cc:492] Webserver started at http://127.0.104.62:42611/ using document root <none> and password file <none>
I20251027 14:51:05.912552 3410 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:05.912637 3410 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:05.912966 3410 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:05.915406 3410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "2e70c5f922bd4d3ca24551e61c6b99f1"
format_stamp: "Formatted at 2025-10-27 14:51:05 on dist-test-slave-hxxt"
server_key: "1cef802289cebe331702a5d330d63993"
server_key_iv: "6a0ca4a11cf04cb8a37eed990592d173"
server_key_version: "encryptionkey@0"
I20251027 14:51:05.916113 3410 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "2e70c5f922bd4d3ca24551e61c6b99f1"
format_stamp: "Formatted at 2025-10-27 14:51:05 on dist-test-slave-hxxt"
server_key: "1cef802289cebe331702a5d330d63993"
server_key_iv: "6a0ca4a11cf04cb8a37eed990592d173"
server_key_version: "encryptionkey@0"
I20251027 14:51:05.921129 3410 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:51:05.924154 3425 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:05.925372 3410 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20251027 14:51:05.925518 3410 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "2e70c5f922bd4d3ca24551e61c6b99f1"
format_stamp: "Formatted at 2025-10-27 14:51:05 on dist-test-slave-hxxt"
server_key: "1cef802289cebe331702a5d330d63993"
server_key_iv: "6a0ca4a11cf04cb8a37eed990592d173"
server_key_version: "encryptionkey@0"
I20251027 14:51:05.925699 3410 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:05.952288 3410 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:05.956040 3410 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:05.956305 3410 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:05.974319 3410 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35789
I20251027 14:51:05.974332 3476 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35789 every 8 connection(s)
I20251027 14:51:05.975737 3410 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:05.978142 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3410
I20251027 14:51:05.978338 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:05.978853 416 external_mini_cluster.cc:1468] Setting key 36c5aa08a3e494193d288ff91afc13b9
I20251027 14:51:05.980232 3477 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 27 14:51:05 dist-test-slave-hxxt krb5kdc[3394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576665, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:05.992861 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1: Bootstrap starting.
I20251027 14:51:05.997026 3477 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:05.998366 3477 log.cc:826] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:06.000525 3480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:05.980709 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58624 (local address 127.0.104.62:35789)
1027 14:51:05.981510 (+ 801us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:05.981525 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:05.981581 (+ 56us) server_negotiation.cc:408] Connection header received
1027 14:51:05.983263 (+ 1682us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:05.983295 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:05.983825 (+ 530us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:05.984322 (+ 497us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:05.985722 (+ 1400us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:05.987689 (+ 1967us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:05.988876 (+ 1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:05.989449 (+ 573us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:05.992506 (+ 3057us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:05.992613 (+ 107us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:05.992634 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:05.992679 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:05.996367 (+ 3688us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:05.997063 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:05.997072 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:05.997081 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:05.997210 (+ 129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:05.997512 (+ 302us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:05.997517 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:05.997520 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:05.998132 (+ 612us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:05.998476 (+ 344us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:05.998766 (+ 290us) server_negotiation.cc:300] Negotiation successful
1027 14:51:05.999255 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":359,"threads_started":1}
I20251027 14:51:06.001562 3477 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1: No bootstrap required, opened a new log
I20251027 14:51:06.007095 3477 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35789 } }
I20251027 14:51:06.007419 3477 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:06.007509 3477 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2e70c5f922bd4d3ca24551e61c6b99f1, State: Initialized, Role: FOLLOWER
I20251027 14:51:06.008030 3477 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35789 } }
I20251027 14:51:06.008219 3477 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:06.008327 3477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:06.008488 3477 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:06.010712 3477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35789 } }
I20251027 14:51:06.011181 3477 leader_election.cc:304] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2e70c5f922bd4d3ca24551e61c6b99f1; no voters:
I20251027 14:51:06.011726 3477 leader_election.cc:290] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:06.011831 3482 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:06.012434 3482 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [term 1 LEADER]: Becoming Leader. State: Replica: 2e70c5f922bd4d3ca24551e61c6b99f1, State: Running, Role: LEADER
I20251027 14:51:06.012902 3482 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35789 } }
I20251027 14:51:06.013615 3477 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:06.015689 3484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2e70c5f922bd4d3ca24551e61c6b99f1. Latest consensus state: current_term: 1 leader_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35789 } } }
I20251027 14:51:06.016008 3484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:06.015866 3483 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2e70c5f922bd4d3ca24551e61c6b99f1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35789 } } }
I20251027 14:51:06.016111 3483 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:06.017666 3490 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:06.023160 3490 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:06.031625 3490 catalog_manager.cc:1357] Generated new cluster ID: 4820239a9d5f4904a5d3ce072537996c
I20251027 14:51:06.031739 3490 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:06.074463 3490 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:06.075663 3490 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:06.099946 3490 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2e70c5f922bd4d3ca24551e61c6b99f1: Generated new TSK 0
I20251027 14:51:06.100872 3490 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:06.195175 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35789
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:06.508018 3505 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:06.508293 3505 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:06.508430 3505 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:06.516201 3505 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:06.516297 3505 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:06.516487 3505 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:06.526422 3505 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:06.527844 3505 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:06.529147 3505 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:06.538058 3513 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:06.538120 3510 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:06.538132 3511 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:06.538936 3505 server_base.cc:1047] running on GCE node
I20251027 14:51:06.539448 3505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:06.540313 3505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:06.541548 3505 hybrid_clock.cc:648] HybridClock initialized: now 1761576666541483 us; error 37 us; skew 500 ppm
I20251027 14:51:06.541806 3505 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:06 dist-test-slave-hxxt krb5kdc[3394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:06.544665 3505 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:51:06.546061 3505 webserver.cc:492] Webserver started at http://127.0.104.1:38451/ using document root <none> and password file <none>
I20251027 14:51:06.546756 3505 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:06.546854 3505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:06.547211 3505 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:06.549793 3505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "bdc93e3456af435b8041218be07866c7"
format_stamp: "Formatted at 2025-10-27 14:51:06 on dist-test-slave-hxxt"
server_key: "55be1f9aeb795506ea617581ede939d4"
server_key_iv: "4dc1c0fd12de1da890033c5c406bab63"
server_key_version: "encryptionkey@0"
I20251027 14:51:06.550558 3505 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "bdc93e3456af435b8041218be07866c7"
format_stamp: "Formatted at 2025-10-27 14:51:06 on dist-test-slave-hxxt"
server_key: "55be1f9aeb795506ea617581ede939d4"
server_key_iv: "4dc1c0fd12de1da890033c5c406bab63"
server_key_version: "encryptionkey@0"
I20251027 14:51:06.555496 3505 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:51:06.558600 3520 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:06.559870 3505 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20251027 14:51:06.560058 3505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "bdc93e3456af435b8041218be07866c7"
format_stamp: "Formatted at 2025-10-27 14:51:06 on dist-test-slave-hxxt"
server_key: "55be1f9aeb795506ea617581ede939d4"
server_key_iv: "4dc1c0fd12de1da890033c5c406bab63"
server_key_version: "encryptionkey@0"
I20251027 14:51:06.560250 3505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:06.583742 3505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:06.587757 3505 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:06.588021 3505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:06.589089 3505 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:06.590569 3505 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:06.590655 3505 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:06.590754 3505 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:06.590816 3505 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:06.621369 3505 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35807
I20251027 14:51:06.621469 3632 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35807 every 8 connection(s)
I20251027 14:51:06.622684 3505 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:06.631932 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3505
I20251027 14:51:06.632133 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:06.632786 416 external_mini_cluster.cc:1468] Setting key 7f9435b0c1537f2cc04b5fabc7c313fe
Oct 27 14:51:06 dist-test-slave-hxxt krb5kdc[3394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576666, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:06.641505 3635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:06.625245 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:57471 (local address 127.0.104.62:35789)
1027 14:51:06.625645 (+ 400us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:06.625650 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:06.626556 (+ 906us) server_negotiation.cc:408] Connection header received
1027 14:51:06.628068 (+ 1512us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:06.628075 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:06.628162 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:06.628263 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:06.629998 (+ 1735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:06.630948 (+ 950us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:06.632037 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:06.632559 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:06.635525 (+ 2966us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:06.635557 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:06.635561 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:06.635594 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:06.638512 (+ 2918us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:06.639179 (+ 667us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:06.639185 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:06.639188 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:06.639276 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:06.639634 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:06.639638 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:06.639641 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:06.639891 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:06.640029 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:06.640901 (+ 872us) server_negotiation.cc:300] Negotiation successful
1027 14:51:06.641159 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":215,"threads_started":1}
I20251027 14:51:06.642516 3636 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:06.625715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35789 (local address 127.0.104.1:57471)
1027 14:51:06.626386 (+ 671us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:06.626428 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:06.627774 (+ 1346us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:06.628554 (+ 780us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:06.628562 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:06.629119 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:06.629794 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:06.629814 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:06.631108 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:06.631113 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:06.631895 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:06.631905 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:06.632179 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:06.633027 (+ 848us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:06.633069 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:06.635331 (+ 2262us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:06.638668 (+ 3337us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:06.638675 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:06.638689 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:06.639077 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:06.639420 (+ 343us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:06.639424 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:06.639427 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:06.639540 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:06.640011 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:06.640017 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:06.640496 (+ 479us) client_negotiation.cc:769] Sending connection context
1027 14:51:06.640816 (+ 320us) client_negotiation.cc:240] Negotiation successful
1027 14:51:06.641338 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":453,"thread_start_us":293,"threads_started":1}
I20251027 14:51:06.644029 3633 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35789
I20251027 14:51:06.644352 3633 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:06.645115 3633 heartbeater.cc:507] Master 127.0.104.62:35789 requested a full tablet report, sending...
I20251027 14:51:06.647392 3442 ts_manager.cc:194] Registered new tserver with Master: bdc93e3456af435b8041218be07866c7 (127.0.104.1:35807)
I20251027 14:51:06.649513 3442 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:57471
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:06.707413 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35789
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:07.023690 3641 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:07.023983 3641 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:07.024082 3641 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:07.032418 3641 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:07.032609 3641 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:07.032864 3641 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:07.047739 3641 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:07.049804 3641 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:07.051465 3641 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:07.062568 3646 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:07.063650 3641 server_base.cc:1047] running on GCE node
W20251027 14:51:07.063751 3649 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:07.068449 3647 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:07.069192 3641 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:07.070142 3641 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:07.071437 3641 hybrid_clock.cc:648] HybridClock initialized: now 1761576667071349 us; error 60 us; skew 500 ppm
I20251027 14:51:07.071779 3641 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:07 dist-test-slave-hxxt krb5kdc[3394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:07.075145 3641 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:51:07.076764 3641 webserver.cc:492] Webserver started at http://127.0.104.2:42137/ using document root <none> and password file <none>
I20251027 14:51:07.077502 3641 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:07.077581 3641 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:07.077906 3641 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:07.080508 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "3f2dff8441984a2ab58bf991c4cf3297"
format_stamp: "Formatted at 2025-10-27 14:51:07 on dist-test-slave-hxxt"
server_key: "092b5a7f75204150b13b531608b6147e"
server_key_iv: "07ed9e749eafc68fcca79e2cf9843778"
server_key_version: "encryptionkey@0"
I20251027 14:51:07.081288 3641 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "3f2dff8441984a2ab58bf991c4cf3297"
format_stamp: "Formatted at 2025-10-27 14:51:07 on dist-test-slave-hxxt"
server_key: "092b5a7f75204150b13b531608b6147e"
server_key_iv: "07ed9e749eafc68fcca79e2cf9843778"
server_key_version: "encryptionkey@0"
I20251027 14:51:07.086603 3641 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:51:07.089927 3656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:07.091281 3641 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:51:07.091434 3641 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "3f2dff8441984a2ab58bf991c4cf3297"
format_stamp: "Formatted at 2025-10-27 14:51:07 on dist-test-slave-hxxt"
server_key: "092b5a7f75204150b13b531608b6147e"
server_key_iv: "07ed9e749eafc68fcca79e2cf9843778"
server_key_version: "encryptionkey@0"
I20251027 14:51:07.091641 3641 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:07.105291 3641 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:07.109158 3641 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:07.109395 3641 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:07.110284 3641 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:07.111701 3641 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:07.111783 3641 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:07.111876 3641 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:07.111946 3641 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:07.142518 3641 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34631
I20251027 14:51:07.142585 3768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34631 every 8 connection(s)
I20251027 14:51:07.143829 3641 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:07.144291 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3641
I20251027 14:51:07.144462 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:07.144948 416 external_mini_cluster.cc:1468] Setting key 230170555f0a6b7a9b11793c229c3e54
Oct 27 14:51:07 dist-test-slave-hxxt krb5kdc[3394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:51:07.161872 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:07.146658 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:41113 (local address 127.0.104.62:35789)
1027 14:51:07.147049 (+ 391us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:07.147055 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:07.148234 (+ 1179us) server_negotiation.cc:408] Connection header received
1027 14:51:07.149747 (+ 1513us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:07.149753 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:07.149835 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:07.149937 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:07.151569 (+ 1632us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.152508 (+ 939us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:07.153560 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.153979 (+ 419us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:07.156607 (+ 2628us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:07.156643 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:07.156647 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:07.156677 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:07.159163 (+ 2486us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:07.159792 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:07.159797 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:07.159799 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:07.159873 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:07.160186 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:07.160191 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:07.160194 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:07.160454 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:07.160592 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:07.161280 (+ 688us) server_negotiation.cc:300] Negotiation successful
1027 14:51:07.161560 (+ 280us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":279,"thread_start_us":172,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:51:07.162825 3772 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:07.147330 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35789 (local address 127.0.104.2:41113)
1027 14:51:07.148089 (+ 759us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:07.148124 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:07.149483 (+ 1359us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:07.150188 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:07.150198 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:07.150738 (+ 540us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:07.151404 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:07.151423 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.152645 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:07.152649 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:07.153431 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:07.153441 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.153693 (+ 252us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:07.154432 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:07.154455 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:07.156435 (+ 1980us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:07.159300 (+ 2865us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:07.159308 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:07.159326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:07.159684 (+ 358us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:07.159966 (+ 282us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:07.159970 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:07.159972 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:07.160091 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:07.160564 (+ 473us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:07.160572 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:07.160886 (+ 314us) client_negotiation.cc:769] Sending connection context
1027 14:51:07.161185 (+ 299us) client_negotiation.cc:240] Negotiation successful
1027 14:51:07.161631 (+ 446us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":343,"threads_started":1}
I20251027 14:51:07.164191 3769 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35789
I20251027 14:51:07.164630 3769 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:07.165520 3769 heartbeater.cc:507] Master 127.0.104.62:35789 requested a full tablet report, sending...
I20251027 14:51:07.167150 3442 ts_manager.cc:194] Registered new tserver with Master: 3f2dff8441984a2ab58bf991c4cf3297 (127.0.104.2:34631)
I20251027 14:51:07.167905 3442 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:41113
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:07.217067 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35789
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:07.525053 3777 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:07.525337 3777 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:07.525439 3777 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:07.533381 3777 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:07.533478 3777 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:07.533622 3777 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:07.543860 3777 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35789
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:07.545361 3777 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:07.546614 3777 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:07.555378 3782 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:07.555351 3783 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:07.555754 3785 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:07.556361 3777 server_base.cc:1047] running on GCE node
I20251027 14:51:07.556903 3777 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:07.557766 3777 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:07.558979 3777 hybrid_clock.cc:648] HybridClock initialized: now 1761576667558921 us; error 33 us; skew 500 ppm
I20251027 14:51:07.559254 3777 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:07 dist-test-slave-hxxt krb5kdc[3394](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:07.562263 3777 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:51:07.563709 3777 webserver.cc:492] Webserver started at http://127.0.104.3:42277/ using document root <none> and password file <none>
I20251027 14:51:07.564448 3777 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:07.564539 3777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:07.564886 3777 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:07.567445 3777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "009905d6c932401f9e260f20d7439658"
format_stamp: "Formatted at 2025-10-27 14:51:07 on dist-test-slave-hxxt"
server_key: "304faa911815e42cc7f686c60a079d3a"
server_key_iv: "f926e74dc04c1000cf259de4bbc8099a"
server_key_version: "encryptionkey@0"
I20251027 14:51:07.568166 3777 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "009905d6c932401f9e260f20d7439658"
format_stamp: "Formatted at 2025-10-27 14:51:07 on dist-test-slave-hxxt"
server_key: "304faa911815e42cc7f686c60a079d3a"
server_key_iv: "f926e74dc04c1000cf259de4bbc8099a"
server_key_version: "encryptionkey@0"
I20251027 14:51:07.573236 3777 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:51:07.576440 3792 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:07.577651 3777 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:51:07.577792 3777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "009905d6c932401f9e260f20d7439658"
format_stamp: "Formatted at 2025-10-27 14:51:07 on dist-test-slave-hxxt"
server_key: "304faa911815e42cc7f686c60a079d3a"
server_key_iv: "f926e74dc04c1000cf259de4bbc8099a"
server_key_version: "encryptionkey@0"
I20251027 14:51:07.577975 3777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:07.593077 3777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:07.599860 3777 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:07.600147 3777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:07.601212 3777 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:07.602752 3777 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:07.602851 3777 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:07.602962 3777 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:07.603031 3777 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:07.633689 3777 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46475
I20251027 14:51:07.633747 3904 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46475 every 8 connection(s)
I20251027 14:51:07.635094 3777 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:07.644167 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3777
I20251027 14:51:07.644341 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRebalanceReportUnauthorized.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:07.644815 416 external_mini_cluster.cc:1468] Setting key 1a6580bb323fce06eddcacec202db710
Oct 27 14:51:07 dist-test-slave-hxxt krb5kdc[3394](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576667, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:07.652634 3633 heartbeater.cc:499] Master 127.0.104.62:35789 was elected leader, sending a full tablet report...
I20251027 14:51:07.653065 3771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:07.637858 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:49283 (local address 127.0.104.62:35789)
1027 14:51:07.638050 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:07.638057 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:07.639269 (+ 1212us) server_negotiation.cc:408] Connection header received
1027 14:51:07.640772 (+ 1503us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:07.640778 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:07.640847 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:07.640945 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:07.642503 (+ 1558us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.643391 (+ 888us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:07.644540 (+ 1149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.645026 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:07.647902 (+ 2876us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:07.647932 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:07.647936 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:07.647967 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:07.650411 (+ 2444us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:07.651064 (+ 653us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:07.651074 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:07.651077 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:07.651153 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:07.651450 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:07.651453 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:07.651455 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:07.651669 (+ 214us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:07.651778 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:07.652580 (+ 802us) server_negotiation.cc:300] Negotiation successful
1027 14:51:07.652790 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:51:07.654182 3907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:07.638385 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35789 (local address 127.0.104.3:49283)
1027 14:51:07.639106 (+ 721us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:07.639145 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:07.640508 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:07.641137 (+ 629us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:07.641145 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:07.641673 (+ 528us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:07.642337 (+ 664us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:07.642354 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.643545 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:07.643549 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:07.644365 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:07.644418 (+ 53us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:07.644626 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:07.645286 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:07.645310 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:07.647683 (+ 2373us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:07.650539 (+ 2856us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:07.650546 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:07.650559 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:07.650943 (+ 384us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:07.651262 (+ 319us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:07.651265 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:07.651268 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:07.651373 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:07.651814 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:07.651821 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:07.652129 (+ 308us) client_negotiation.cc:769] Sending connection context
1027 14:51:07.652500 (+ 371us) client_negotiation.cc:240] Negotiation successful
1027 14:51:07.653045 (+ 545us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":492,"thread_start_us":312,"threads_started":1}
I20251027 14:51:07.655392 3905 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35789
I20251027 14:51:07.655689 3905 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:07.656442 3905 heartbeater.cc:507] Master 127.0.104.62:35789 requested a full tablet report, sending...
I20251027 14:51:07.657788 3442 ts_manager.cc:194] Registered new tserver with Master: 009905d6c932401f9e260f20d7439658 (127.0.104.3:46475)
I20251027 14:51:07.658681 3442 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:49283
I20251027 14:51:07.660454 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251027 14:51:08.013769 3771 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:08.008346 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58632 (local address 127.0.104.62:35789)
1027 14:51:08.008529 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:08.008535 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:08.010078 (+ 1543us) server_negotiation.cc:408] Connection header received
1027 14:51:08.011721 (+ 1643us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:08.011728 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:08.011812 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:08.011902 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:08.013515 (+ 1613us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58632: BlockingRecv error: recv got EOF from 127.0.0.1:58632 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
W20251027 14:51:08.015856 3771 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:08.014665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58646 (local address 127.0.104.62:35789)
1027 14:51:08.014860 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:08.014866 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:08.014883 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:51:08.014924 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:08.014928 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:08.014980 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:08.015066 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:08.015655 (+ 589us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58646: BlockingRecv error: recv got EOF from 127.0.0.1:58646 (error 108)
Metrics: {"server-negotiator.queue_time_us":48}
W20251027 14:51:08.017083 3771 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:08.016104 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58652 (local address 127.0.104.62:35789)
1027 14:51:08.016230 (+ 126us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:08.016233 (+ 3us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:08.016247 (+ 14us) server_negotiation.cc:408] Connection header received
1027 14:51:08.016287 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:08.016291 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:08.016341 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:08.016452 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:08.016897 (+ 445us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58652: BlockingRecv error: recv got EOF from 127.0.0.1:58652 (error 108)
Metrics: {"server-negotiator.queue_time_us":39}
W20251027 14:51:08.023531 3771 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:08.021905 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58660 (local address 127.0.104.62:35789)
1027 14:51:08.022088 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:08.022095 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:08.022412 (+ 317us) server_negotiation.cc:408] Connection header received
1027 14:51:08.022526 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:08.022530 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:08.022585 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:08.022659 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:08.023343 (+ 684us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58660: BlockingRecv error: recv got EOF from 127.0.0.1:58660 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:51:08.102202 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3505
W20251027 14:51:08.105510 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3505: Not authorized: Client connection negotiation failed: client connection to 127.0.104.1:35807: server requires authentication, but client does not have Kerberos credentials available
I20251027 14:51:08.105738 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3505
W20251027 14:51:08.105739 3928 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:08.103281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58878 (local address 127.0.104.1:35807)
1027 14:51:08.103980 (+ 699us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:08.103988 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:08.104015 (+ 27us) server_negotiation.cc:408] Connection header received
1027 14:51:08.104096 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:08.104104 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:08.104345 (+ 241us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:08.104564 (+ 219us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:08.105464 (+ 900us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58878: BlockingRecv error: recv got EOF from 127.0.0.1:58878 (error 108)
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":436,"threads_started":1}
I20251027 14:51:08.118336 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3641
W20251027 14:51:08.120883 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3641: Not authorized: Client connection negotiation failed: client connection to 127.0.104.2:34631: server requires authentication, but client does not have Kerberos credentials available
I20251027 14:51:08.121007 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3641
I20251027 14:51:08.133370 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3777
W20251027 14:51:08.135839 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3777: Not authorized: Client connection negotiation failed: client connection to 127.0.104.3:46475: server requires authentication, but client does not have Kerberos credentials available
I20251027 14:51:08.136000 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3777
I20251027 14:51:08.148180 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3410
I20251027 14:51:08.149484 3472 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:08.250793 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3410
2025-10-27T14:51:08Z chronyd exiting
[ OK ] SecurityITest.TestRebalanceReportUnauthorized (4755 ms)
[ RUN ] SecurityITest.SaslPlainFallback
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:08 dist-test-slave-hxxt krb5kdc[3934](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:08 dist-test-slave-hxxt krb5kdc[3934](info): set up 2 sockets
Oct 27 14:51:08 dist-test-slave-hxxt krb5kdc[3934](info): commencing operation
krb5kdc: starting...
W20251027 14:51:10.112565 416 mini_kdc.cc:121] Time spent starting KDC: real 1.824s user 0.004s sys 0.012s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:10 dist-test-slave-hxxt krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:10.328845 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46751
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:46751
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc-authentication=optional
--user-acl=* with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:10.643826 3950 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:10.644160 3950 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:10.644289 3950 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:10.652163 3950 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:10.652269 3950 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:10.652349 3950 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:10.652426 3950 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:10.652503 3950 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:10.662765 3950 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:46751
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:10.664299 3950 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:10.665576 3950 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:10.673262 3955 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:10.673357 3956 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:10.674005 3958 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:10.674311 3950 server_base.cc:1047] running on GCE node
I20251027 14:51:10.675076 3950 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:10.676405 3950 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:10.677647 3950 hybrid_clock.cc:648] HybridClock initialized: now 1761576670677572 us; error 45 us; skew 500 ppm
I20251027 14:51:10.678103 3950 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:10 dist-test-slave-hxxt krb5kdc[3934](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576670, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:10.681048 3950 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:10.682578 3950 webserver.cc:492] Webserver started at http://127.0.104.62:37249/ using document root <none> and password file <none>
I20251027 14:51:10.683300 3950 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:10.683384 3950 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:10.683723 3950 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:10.686226 3950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "26d2fcf44c86475fa01d9fe60d289c1c"
format_stamp: "Formatted at 2025-10-27 14:51:10 on dist-test-slave-hxxt"
server_key: "6b450fa348dad83e67efac35c3f5f7d6"
server_key_iv: "9963b2e187d8545c47bd8f5c1d1986fe"
server_key_version: "encryptionkey@0"
I20251027 14:51:10.686978 3950 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "26d2fcf44c86475fa01d9fe60d289c1c"
format_stamp: "Formatted at 2025-10-27 14:51:10 on dist-test-slave-hxxt"
server_key: "6b450fa348dad83e67efac35c3f5f7d6"
server_key_iv: "9963b2e187d8545c47bd8f5c1d1986fe"
server_key_version: "encryptionkey@0"
I20251027 14:51:10.691989 3950 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:51:10.695240 3965 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:10.696597 3950 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:51:10.696751 3950 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "26d2fcf44c86475fa01d9fe60d289c1c"
format_stamp: "Formatted at 2025-10-27 14:51:10 on dist-test-slave-hxxt"
server_key: "6b450fa348dad83e67efac35c3f5f7d6"
server_key_iv: "9963b2e187d8545c47bd8f5c1d1986fe"
server_key_version: "encryptionkey@0"
I20251027 14:51:10.696960 3950 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:10.715620 3950 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:10.719236 3950 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:10.719467 3950 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:10.737497 3950 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46751
I20251027 14:51:10.737517 4016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46751 every 8 connection(s)
I20251027 14:51:10.738920 3950 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:10.742895 4017 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:10.745528 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 3950
I20251027 14:51:10.745748 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.SaslPlainFallback.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:10.746277 416 external_mini_cluster.cc:1468] Setting key 416f258962f0f2144dc5861fe9dfddfc
I20251027 14:51:10.754472 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c: Bootstrap starting.
I20251027 14:51:10.757782 4017 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c: Neither blocks nor log segments found. Creating new log.
Oct 27 14:51:10 dist-test-slave-hxxt krb5kdc[3934](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576670, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:10.759017 4017 log.cc:826] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:10.762306 4017 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c: No bootstrap required, opened a new log
I20251027 14:51:10.767715 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:10.748092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55272 (local address 127.0.104.62:46751)
1027 14:51:10.748883 (+ 791us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:10.748899 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:10.748963 (+ 64us) server_negotiation.cc:408] Connection header received
1027 14:51:10.750465 (+ 1502us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:10.750493 (+ 28us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:10.751032 (+ 539us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:10.751517 (+ 485us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:10.752904 (+ 1387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:10.754793 (+ 1889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:10.755880 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:10.756456 (+ 576us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:10.759649 (+ 3193us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:10.759752 (+ 103us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:10.759778 (+ 26us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:10.759826 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:10.762886 (+ 3060us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:10.763769 (+ 883us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:10.763779 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:10.763788 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:10.763909 (+ 121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:10.764319 (+ 410us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:10.764324 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:10.764327 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:10.764999 (+ 672us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:10.765330 (+ 331us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:10.765914 (+ 584us) server_negotiation.cc:300] Negotiation successful
1027 14:51:10.766379 (+ 465us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":545,"thread_start_us":351,"threads_started":1}
I20251027 14:51:10.769217 4017 raft_consensus.cc:359] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46751 } }
I20251027 14:51:10.769547 4017 raft_consensus.cc:385] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:10.769644 4017 raft_consensus.cc:740] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 26d2fcf44c86475fa01d9fe60d289c1c, State: Initialized, Role: FOLLOWER
I20251027 14:51:10.770237 4017 consensus_queue.cc:260] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46751 } }
I20251027 14:51:10.770442 4017 raft_consensus.cc:399] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:10.770605 4017 raft_consensus.cc:493] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:10.770757 4017 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:10.772074 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:51:10.773145 4017 raft_consensus.cc:515] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46751 } }
I20251027 14:51:10.773576 4017 leader_election.cc:304] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 26d2fcf44c86475fa01d9fe60d289c1c; no voters:
I20251027 14:51:10.774195 4017 leader_election.cc:290] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:10.774327 4022 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:10.775926 4017 sys_catalog.cc:565] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:10.779970 4022 raft_consensus.cc:697] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [term 1 LEADER]: Becoming Leader. State: Replica: 26d2fcf44c86475fa01d9fe60d289c1c, State: Running, Role: LEADER
I20251027 14:51:10.780539 4022 consensus_queue.cc:237] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46751 } }
I20251027 14:51:10.783493 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 26d2fcf44c86475fa01d9fe60d289c1c. Latest consensus state: current_term: 1 leader_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46751 } } }
I20251027 14:51:10.783511 4023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "26d2fcf44c86475fa01d9fe60d289c1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46751 } } }
I20251027 14:51:10.783854 4023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:10.783856 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:10.789799 4033 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:10.794986 4033 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:10.807575 4033 catalog_manager.cc:1357] Generated new cluster ID: 3aae1d35773842f991ee77d6c37cc479
I20251027 14:51:10.807793 4033 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:10.815289 4020 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:10.807209 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59498 (local address 127.0.104.62:46751)
1027 14:51:10.807360 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:10.807365 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:10.807888 (+ 523us) server_negotiation.cc:408] Connection header received
1027 14:51:10.808174 (+ 286us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:10.808181 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:10.808273 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:10.808409 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:10.809903 (+ 1494us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:10.811261 (+ 1358us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:10.812827 (+ 1566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:10.813421 (+ 594us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:10.813648 (+ 227us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:10.813681 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:10.813687 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:10.814485 (+ 798us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:10.814559 (+ 74us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:10.814588 (+ 29us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:10.814697 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:10.814826 (+ 129us) server_negotiation.cc:300] Negotiation successful
1027 14:51:10.814943 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251027 14:51:10.832607 4033 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:10.834376 4033 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:10.846395 4033 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 26d2fcf44c86475fa01d9fe60d289c1c: Generated new TSK 0
I20251027 14:51:10.847230 4033 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:51:10.861117 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 3950
I20251027 14:51:10.863577 4012 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:10.951696 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 3950
2025-10-27T14:51:10Z chronyd exiting
[ OK ] SecurityITest.SaslPlainFallback (2695 ms)
[ RUN ] SecurityITest.TestUnauthorizedClientKerberosCredentials
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:11 dist-test-slave-hxxt krb5kdc[4051](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:11 dist-test-slave-hxxt krb5kdc[4051](info): set up 2 sockets
Oct 27 14:51:11 dist-test-slave-hxxt krb5kdc[4051](info): commencing operation
krb5kdc: starting...
W20251027 14:51:12.820669 416 mini_kdc.cc:121] Time spent starting KDC: real 1.836s user 0.002s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:12 dist-test-slave-hxxt krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:13.028362 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34559
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34559
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:13.336328 4067 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:13.336678 4067 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:13.336822 4067 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:13.344739 4067 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:13.344851 4067 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:13.344930 4067 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:13.344992 4067 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:13.345073 4067 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:13.355343 4067 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34559
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34559
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:13.356925 4067 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:13.358191 4067 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:13.366063 4073 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:13.366108 4072 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:13.366542 4075 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:13.367026 4067 server_base.cc:1047] running on GCE node
I20251027 14:51:13.367787 4067 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:13.369194 4067 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:13.370409 4067 hybrid_clock.cc:648] HybridClock initialized: now 1761576673370345 us; error 47 us; skew 500 ppm
I20251027 14:51:13.370723 4067 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:13 dist-test-slave-hxxt krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:13.373684 4067 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:13.375236 4067 webserver.cc:492] Webserver started at http://127.0.104.62:34143/ using document root <none> and password file <none>
I20251027 14:51:13.375943 4067 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:13.376027 4067 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:13.376366 4067 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:13.378944 4067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "2099d2f53b8346d3b1be056d23c91958"
format_stamp: "Formatted at 2025-10-27 14:51:13 on dist-test-slave-hxxt"
server_key: "44e2899fa9053105b9b8224e88247bf4"
server_key_iv: "b9317be0bbcb948cc7b2db14bc2bfd3e"
server_key_version: "encryptionkey@0"
I20251027 14:51:13.379700 4067 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "2099d2f53b8346d3b1be056d23c91958"
format_stamp: "Formatted at 2025-10-27 14:51:13 on dist-test-slave-hxxt"
server_key: "44e2899fa9053105b9b8224e88247bf4"
server_key_iv: "b9317be0bbcb948cc7b2db14bc2bfd3e"
server_key_version: "encryptionkey@0"
I20251027 14:51:13.384959 4067 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20251027 14:51:13.388207 4082 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:13.389521 4067 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:13.389690 4067 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "2099d2f53b8346d3b1be056d23c91958"
format_stamp: "Formatted at 2025-10-27 14:51:13 on dist-test-slave-hxxt"
server_key: "44e2899fa9053105b9b8224e88247bf4"
server_key_iv: "b9317be0bbcb948cc7b2db14bc2bfd3e"
server_key_version: "encryptionkey@0"
I20251027 14:51:13.389899 4067 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:13.420675 4067 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:13.427358 4067 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:13.427625 4067 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:13.445751 4067 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34559
I20251027 14:51:13.445771 4133 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34559 every 8 connection(s)
I20251027 14:51:13.447167 4067 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:13.451102 4134 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:13.453892 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4067
I20251027 14:51:13.454121 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:13.454806 416 external_mini_cluster.cc:1468] Setting key 6ec8a3b5832f1b2f93920864a20e51de
I20251027 14:51:13.462836 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958: Bootstrap starting.
I20251027 14:51:13.466835 4134 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958: Neither blocks nor log segments found. Creating new log.
Oct 27 14:51:13 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576672, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:13.467851 4134 log.cc:826] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:13.471215 4134 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958: No bootstrap required, opened a new log
I20251027 14:51:13.475896 4137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:13.456786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44196 (local address 127.0.104.62:34559)
1027 14:51:13.457508 (+ 722us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:13.457520 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:13.457561 (+ 41us) server_negotiation.cc:408] Connection header received
1027 14:51:13.458718 (+ 1157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:13.458741 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:13.459162 (+ 421us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:13.459551 (+ 389us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:13.461408 (+ 1857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:13.462724 (+ 1316us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:13.464132 (+ 1408us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:13.464685 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:13.468245 (+ 3560us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:13.468328 (+ 83us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:13.468351 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:13.468420 (+ 69us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:13.471895 (+ 3475us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:13.472530 (+ 635us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:13.472539 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:13.472547 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:13.472658 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:13.472948 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:13.472954 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:13.472957 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:13.473560 (+ 603us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:13.473849 (+ 289us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:13.474157 (+ 308us) server_negotiation.cc:300] Negotiation successful
1027 14:51:13.474619 (+ 462us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":515,"thread_start_us":343,"threads_started":1}
I20251027 14:51:13.477803 4134 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2099d2f53b8346d3b1be056d23c91958" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34559 } }
I20251027 14:51:13.478106 4134 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:13.478205 4134 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2099d2f53b8346d3b1be056d23c91958, State: Initialized, Role: FOLLOWER
I20251027 14:51:13.478786 4134 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2099d2f53b8346d3b1be056d23c91958" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34559 } }
I20251027 14:51:13.478971 4134 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:13.479080 4134 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:13.479215 4134 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:13.481391 4134 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2099d2f53b8346d3b1be056d23c91958" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34559 } }
I20251027 14:51:13.481841 4134 leader_election.cc:304] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2099d2f53b8346d3b1be056d23c91958; no voters:
I20251027 14:51:13.482457 4134 leader_election.cc:290] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:13.482601 4139 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:13.483325 4139 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [term 1 LEADER]: Becoming Leader. State: Replica: 2099d2f53b8346d3b1be056d23c91958, State: Running, Role: LEADER
I20251027 14:51:13.483798 4139 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2099d2f53b8346d3b1be056d23c91958" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34559 } }
I20251027 14:51:13.484347 4134 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:13.486455 4141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2099d2f53b8346d3b1be056d23c91958. Latest consensus state: current_term: 1 leader_uuid: "2099d2f53b8346d3b1be056d23c91958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2099d2f53b8346d3b1be056d23c91958" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34559 } } }
I20251027 14:51:13.486809 4141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:13.486864 4140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2099d2f53b8346d3b1be056d23c91958" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2099d2f53b8346d3b1be056d23c91958" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34559 } } }
I20251027 14:51:13.487440 4140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:13.489315 4148 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:13.493755 4148 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:13.502144 4148 catalog_manager.cc:1357] Generated new cluster ID: 8397fbdd136f4f6e87b993e6634772d0
I20251027 14:51:13.502274 4148 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:13.524209 4148 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:13.525460 4148 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:13.544246 4148 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2099d2f53b8346d3b1be056d23c91958: Generated new TSK 0
I20251027 14:51:13.545132 4148 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:13.638980 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34559
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:13.955020 4162 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:13.955281 4162 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:13.955387 4162 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:13.963193 4162 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:13.963289 4162 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:13.963450 4162 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:13.973510 4162 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:13.975039 4162 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:13.976343 4162 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:13.985271 4168 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:13.985390 4167 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:13.985344 4170 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:13.986697 4162 server_base.cc:1047] running on GCE node
I20251027 14:51:13.987197 4162 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:13.988157 4162 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:13.989388 4162 hybrid_clock.cc:648] HybridClock initialized: now 1761576673989309 us; error 62 us; skew 500 ppm
I20251027 14:51:13.989651 4162 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:13 dist-test-slave-hxxt krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:13.992714 4162 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:51:13.994293 4162 webserver.cc:492] Webserver started at http://127.0.104.1:39595/ using document root <none> and password file <none>
I20251027 14:51:13.995069 4162 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:13.995144 4162 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:13.995478 4162 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:13.998090 4162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "05e28d34dccd4283b0f3bbd7cf8a0552"
format_stamp: "Formatted at 2025-10-27 14:51:13 on dist-test-slave-hxxt"
server_key: "21dc4b2fef52bdd2fb9842a1f8889389"
server_key_iv: "90877904672f97c16101b1d531be947e"
server_key_version: "encryptionkey@0"
I20251027 14:51:13.998809 4162 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "05e28d34dccd4283b0f3bbd7cf8a0552"
format_stamp: "Formatted at 2025-10-27 14:51:13 on dist-test-slave-hxxt"
server_key: "21dc4b2fef52bdd2fb9842a1f8889389"
server_key_iv: "90877904672f97c16101b1d531be947e"
server_key_version: "encryptionkey@0"
I20251027 14:51:14.003782 4162 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:51:14.007066 4177 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:14.008301 4162 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:51:14.008508 4162 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "05e28d34dccd4283b0f3bbd7cf8a0552"
format_stamp: "Formatted at 2025-10-27 14:51:13 on dist-test-slave-hxxt"
server_key: "21dc4b2fef52bdd2fb9842a1f8889389"
server_key_iv: "90877904672f97c16101b1d531be947e"
server_key_version: "encryptionkey@0"
I20251027 14:51:14.008701 4162 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:14.025020 4162 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:14.028930 4162 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:14.029160 4162 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:14.030053 4162 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:14.031474 4162 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:14.031565 4162 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:14.031636 4162 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:14.031690 4162 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:14.061621 4162 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39077
I20251027 14:51:14.061668 4289 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39077 every 8 connection(s)
I20251027 14:51:14.062914 4162 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:14.066247 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4162
I20251027 14:51:14.066512 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:14.067176 416 external_mini_cluster.cc:1468] Setting key 0bf66105c57897f8d1b2688bd2a2b9a3
Oct 27 14:51:14 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576673, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:14.082710 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:14.065428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:50845 (local address 127.0.104.62:34559)
1027 14:51:14.065800 (+ 372us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:14.065805 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:14.066974 (+ 1169us) server_negotiation.cc:408] Connection header received
1027 14:51:14.068647 (+ 1673us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:14.068654 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:14.068741 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:14.068858 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:14.071076 (+ 2218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.071967 (+ 891us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:14.073399 (+ 1432us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.073727 (+ 328us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:14.076797 (+ 3070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:14.076833 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:14.076838 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:14.076886 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:14.079761 (+ 2875us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:14.080504 (+ 743us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:14.080510 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:14.080513 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:14.080596 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:14.080921 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:14.080926 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:14.080929 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:14.081225 (+ 296us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:14.081377 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:14.082082 (+ 705us) server_negotiation.cc:300] Negotiation successful
1027 14:51:14.082348 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":275,"thread_start_us":179,"threads_started":1}
I20251027 14:51:14.083473 4293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:14.066032 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34559 (local address 127.0.104.1:50845)
1027 14:51:14.066809 (+ 777us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:14.066847 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:14.068322 (+ 1475us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:14.069136 (+ 814us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:14.069149 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:14.069931 (+ 782us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:14.070882 (+ 951us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:14.070909 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.072117 (+ 1208us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:14.072122 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:14.073253 (+ 1131us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:14.073279 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.073545 (+ 266us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:14.074239 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:14.074268 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:14.076592 (+ 2324us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:14.079918 (+ 3326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:14.079926 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:14.079941 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:14.080328 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:14.080703 (+ 375us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:14.080707 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:14.080710 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:14.080819 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:14.081330 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:14.081336 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:14.081659 (+ 323us) client_negotiation.cc:769] Sending connection context
1027 14:51:14.081942 (+ 283us) client_negotiation.cc:240] Negotiation successful
1027 14:51:14.082378 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":533,"thread_start_us":344,"threads_started":1}
I20251027 14:51:14.084987 4290 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34559
I20251027 14:51:14.085338 4290 heartbeater.cc:461] Registering TS with master...
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:51:14.086162 4290 heartbeater.cc:507] Master 127.0.104.62:34559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:51:14.088562 4099 ts_manager.cc:194] Registered new tserver with Master: 05e28d34dccd4283b0f3bbd7cf8a0552 (127.0.104.1:39077)
I20251027 14:51:14.090570 4099 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:50845
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:14.142812 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34559
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:14.460737 4298 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:14.461117 4298 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:14.461268 4298 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:14.472002 4298 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:14.472103 4298 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:14.472272 4298 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:14.482489 4298 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:14.483961 4298 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:14.485285 4298 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:14.494439 4304 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:14.494469 4303 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:14.494829 4306 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:14.495548 4298 server_base.cc:1047] running on GCE node
I20251027 14:51:14.496074 4298 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:14.496979 4298 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:14.498178 4298 hybrid_clock.cc:648] HybridClock initialized: now 1761576674498098 us; error 49 us; skew 500 ppm
I20251027 14:51:14.498440 4298 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:14 dist-test-slave-hxxt krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:14.501395 4298 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:51:14.502889 4298 webserver.cc:492] Webserver started at http://127.0.104.2:33121/ using document root <none> and password file <none>
I20251027 14:51:14.503592 4298 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:14.503681 4298 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:14.504011 4298 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:14.506563 4298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "882b8ba43c1e425fa7dae28d85ffe693"
format_stamp: "Formatted at 2025-10-27 14:51:14 on dist-test-slave-hxxt"
server_key: "9cacdb6370fb8ee70e5cb1eb88b1f8b4"
server_key_iv: "1a4d21d5ff15ae92b6745e227407c99c"
server_key_version: "encryptionkey@0"
I20251027 14:51:14.507303 4298 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "882b8ba43c1e425fa7dae28d85ffe693"
format_stamp: "Formatted at 2025-10-27 14:51:14 on dist-test-slave-hxxt"
server_key: "9cacdb6370fb8ee70e5cb1eb88b1f8b4"
server_key_iv: "1a4d21d5ff15ae92b6745e227407c99c"
server_key_version: "encryptionkey@0"
I20251027 14:51:14.512307 4298 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251027 14:51:14.515515 4313 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:14.516796 4298 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20251027 14:51:14.516942 4298 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "882b8ba43c1e425fa7dae28d85ffe693"
format_stamp: "Formatted at 2025-10-27 14:51:14 on dist-test-slave-hxxt"
server_key: "9cacdb6370fb8ee70e5cb1eb88b1f8b4"
server_key_iv: "1a4d21d5ff15ae92b6745e227407c99c"
server_key_version: "encryptionkey@0"
I20251027 14:51:14.517127 4298 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:14.537700 4298 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:14.541231 4298 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:14.541469 4298 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:14.542344 4298 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:14.543768 4298 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:14.543851 4298 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:14.543947 4298 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:14.544011 4298 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:14.574062 4298 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:36987
I20251027 14:51:14.574127 4425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:36987 every 8 connection(s)
I20251027 14:51:14.575359 4298 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:14.579675 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4298
I20251027 14:51:14.579882 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:14.580241 416 external_mini_cluster.cc:1468] Setting key b686f1495ad1a4cd24769bc1a29bd29e
Oct 27 14:51:14 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576674, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:14.595053 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:14.578116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:44631 (local address 127.0.104.62:34559)
1027 14:51:14.578263 (+ 147us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:14.578268 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:14.579502 (+ 1234us) server_negotiation.cc:408] Connection header received
1027 14:51:14.581239 (+ 1737us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:14.581246 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:14.581334 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:14.581430 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:14.583330 (+ 1900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.584281 (+ 951us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:14.585488 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.585952 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:14.588951 (+ 2999us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:14.588991 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:14.588995 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:14.589030 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:14.591661 (+ 2631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:14.592625 (+ 964us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:14.592630 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:14.592632 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:14.592715 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:14.593093 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:14.593096 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:14.593098 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:14.593327 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:14.593451 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:14.594584 (+ 1133us) server_negotiation.cc:300] Negotiation successful
1027 14:51:14.594795 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251027 14:51:14.596587 4428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:14.578638 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34559 (local address 127.0.104.2:44631)
1027 14:51:14.579329 (+ 691us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:14.579377 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:14.580914 (+ 1537us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:14.581696 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:14.581706 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:14.582423 (+ 717us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:14.583147 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:14.583168 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.584520 (+ 1352us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:14.584525 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:14.585353 (+ 828us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:14.585365 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:14.585591 (+ 226us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:14.586256 (+ 665us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:14.586282 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:14.588647 (+ 2365us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:14.591845 (+ 3198us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:14.591854 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:14.591875 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:14.592470 (+ 595us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:14.592827 (+ 357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:14.592832 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:14.592835 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:14.592993 (+ 158us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:14.593523 (+ 530us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:14.593536 (+ 13us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:14.593984 (+ 448us) client_negotiation.cc:769] Sending connection context
1027 14:51:14.594483 (+ 499us) client_negotiation.cc:240] Negotiation successful
1027 14:51:14.595052 (+ 569us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":306,"threads_started":1}
I20251027 14:51:14.598104 4426 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34559
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:51:14.598524 4426 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:14.599406 4426 heartbeater.cc:507] Master 127.0.104.62:34559 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:51:14.600977 4099 ts_manager.cc:194] Registered new tserver with Master: 882b8ba43c1e425fa7dae28d85ffe693 (127.0.104.2:36987)
I20251027 14:51:14.601783 4099 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:44631
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:14.655962 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34559
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:14.969132 4433 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:14.969430 4433 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:14.969558 4433 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:14.977598 4433 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:14.977702 4433 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:14.977900 4433 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:14.988257 4433 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:34559
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:14.989897 4433 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:14.991178 4433 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:15.000492 4438 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:15.000517 4439 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:15.001233 4441 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:15.001683 4433 server_base.cc:1047] running on GCE node
I20251027 14:51:15.002236 4433 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:15.003103 4433 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:15.004324 4433 hybrid_clock.cc:648] HybridClock initialized: now 1761576675004260 us; error 41 us; skew 500 ppm
I20251027 14:51:15.004622 4433 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:15.007647 4433 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:51:15.009123 4433 webserver.cc:492] Webserver started at http://127.0.104.3:35503/ using document root <none> and password file <none>
I20251027 14:51:15.009832 4433 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:15.009907 4433 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:15.010236 4433 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:15.012939 4433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "1b7d946d544544738d6bd44a463a62b5"
format_stamp: "Formatted at 2025-10-27 14:51:15 on dist-test-slave-hxxt"
server_key: "fc4f6bef0e62c19c77b5ec4d75bfb76d"
server_key_iv: "266b0070b10fa7a3c18a952f4039cd70"
server_key_version: "encryptionkey@0"
I20251027 14:51:15.013729 4433 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "1b7d946d544544738d6bd44a463a62b5"
format_stamp: "Formatted at 2025-10-27 14:51:15 on dist-test-slave-hxxt"
server_key: "fc4f6bef0e62c19c77b5ec4d75bfb76d"
server_key_iv: "266b0070b10fa7a3c18a952f4039cd70"
server_key_version: "encryptionkey@0"
I20251027 14:51:15.018918 4433 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:51:15.022137 4448 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:15.023370 4433 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:51:15.023520 4433 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "1b7d946d544544738d6bd44a463a62b5"
format_stamp: "Formatted at 2025-10-27 14:51:15 on dist-test-slave-hxxt"
server_key: "fc4f6bef0e62c19c77b5ec4d75bfb76d"
server_key_iv: "266b0070b10fa7a3c18a952f4039cd70"
server_key_version: "encryptionkey@0"
I20251027 14:51:15.023712 4433 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:15.044150 4433 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:15.048206 4433 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:15.048503 4433 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:15.049417 4433 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:15.050895 4433 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:15.050980 4433 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:15.051082 4433 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:15.051149 4433 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:15.081255 4433 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44807
I20251027 14:51:15.081306 4560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44807 every 8 connection(s)
I20251027 14:51:15.082605 4433 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:15.092836 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4433
I20251027 14:51:15.093048 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestUnauthorizedClientKerberosCredentials.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:15.093526 416 external_mini_cluster.cc:1468] Setting key d66541c52448ebb65d9fc6675f959d47
I20251027 14:51:15.093706 4290 heartbeater.cc:499] Master 127.0.104.62:34559 was elected leader, sending a full tablet report...
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:15.100878 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:15.085116 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:41419 (local address 127.0.104.62:34559)
1027 14:51:15.085262 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:15.085266 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:15.086433 (+ 1167us) server_negotiation.cc:408] Connection header received
1027 14:51:15.087956 (+ 1523us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:15.087963 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:15.088062 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:15.088164 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:15.089889 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.090772 (+ 883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:15.091867 (+ 1095us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.092310 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:15.095991 (+ 3681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:15.096017 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:15.096020 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:15.096048 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:15.098562 (+ 2514us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.099175 (+ 613us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.099178 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.099181 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.099237 (+ 56us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.099479 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.099482 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.099484 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.099694 (+ 210us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:15.099792 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:15.100453 (+ 661us) server_negotiation.cc:300] Negotiation successful
1027 14:51:15.100649 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251027 14:51:15.101745 4563 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:15.085609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34559 (local address 127.0.104.3:41419)
1027 14:51:15.086277 (+ 668us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:15.086313 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:15.087643 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:15.088443 (+ 800us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:15.088452 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:15.089003 (+ 551us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:15.089691 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:15.089711 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.090932 (+ 1221us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:15.090937 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:15.091726 (+ 789us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:15.091737 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.092039 (+ 302us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:15.092987 (+ 948us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:15.093020 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:15.095807 (+ 2787us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:15.098702 (+ 2895us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:15.098709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:15.098723 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:15.099090 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:15.099306 (+ 216us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:15.099315 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:15.099317 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:15.099412 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:15.099782 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:15.099788 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:15.100069 (+ 281us) client_negotiation.cc:769] Sending connection context
1027 14:51:15.100359 (+ 290us) client_negotiation.cc:240] Negotiation successful
1027 14:51:15.100752 (+ 393us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
I20251027 14:51:15.102938 4561 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34559
I20251027 14:51:15.103229 4561 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:15.103929 4561 heartbeater.cc:507] Master 127.0.104.62:34559 requested a full tablet report, sending...
I20251027 14:51:15.105249 4099 ts_manager.cc:194] Registered new tserver with Master: 1b7d946d544544738d6bd44a463a62b5 (127.0.104.3:44807)
I20251027 14:51:15.106041 4099 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:41419
I20251027 14:51:15.108760 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for joe-interloper@KRBTEST.COM:
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:15.151788 4292 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:15.139742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44208 (local address 127.0.104.62:34559)
1027 14:51:15.139925 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:15.139932 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:15.140265 (+ 333us) server_negotiation.cc:408] Connection header received
1027 14:51:15.140459 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:15.140463 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:15.140525 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:15.140613 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:15.142059 (+ 1446us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.142938 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:15.144047 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.144484 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:15.147348 (+ 2864us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:15.147385 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:15.147388 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:15.147421 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:15.149670 (+ 2249us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.150215 (+ 545us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.150220 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.150222 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.150288 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.150563 (+ 275us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.150568 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.150570 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.150815 (+ 245us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:15.151050 (+ 235us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:15.151293 (+ 243us) server_negotiation.cc:300] Negotiation successful
1027 14:51:15.151505 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251027 14:51:15.152297 4099 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.ConnectToMaster from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:44208
I20251027 14:51:15.153991 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4162
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:51:15.166884 4572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:15.154789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43208 (local address 127.0.104.1:39077)
1027 14:51:15.155220 (+ 431us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:15.155227 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:15.155252 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:51:15.155318 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:15.155324 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:15.155527 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:15.155676 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:15.156848 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.158097 (+ 1249us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:15.159172 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.159512 (+ 340us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:15.162378 (+ 2866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:15.162445 (+ 67us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:15.162461 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:15.162490 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:15.164955 (+ 2465us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.165479 (+ 524us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.165486 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.165488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.165557 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.165749 (+ 192us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.165753 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.165754 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.166005 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:15.166227 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:15.166404 (+ 177us) server_negotiation.cc:300] Negotiation successful
1027 14:51:15.166616 (+ 212us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":298,"thread_start_us":223,"threads_started":1}
W20251027 14:51:15.168167 4285 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:43208
W20251027 14:51:15.169253 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4162: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:51:15.169420 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4162
I20251027 14:51:15.182175 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4298
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:51:15.197233 4573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:15.183033 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56754 (local address 127.0.104.2:36987)
1027 14:51:15.183473 (+ 440us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:15.183480 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:15.183505 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:51:15.183580 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:15.183587 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:15.183789 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:15.183965 (+ 176us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:15.185495 (+ 1530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.186686 (+ 1191us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:15.187880 (+ 1194us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.188323 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:15.191534 (+ 3211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:15.191613 (+ 79us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:15.191629 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:15.191663 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:15.194468 (+ 2805us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.195302 (+ 834us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.195311 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.195314 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.195395 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.195733 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.195738 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.195742 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.196037 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:15.196322 (+ 285us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:15.196662 (+ 340us) server_negotiation.cc:300] Negotiation successful
1027 14:51:15.196943 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":314,"thread_start_us":218,"threads_started":1}
W20251027 14:51:15.198499 4421 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:56754
W20251027 14:51:15.199726 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4298: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:51:15.199877 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4298
I20251027 14:51:15.212184 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4433
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4051](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576675, etypes {rep=17 tkt=17 ses=17}, joe-interloper@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:51:15.227321 4574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:15.213109 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59684 (local address 127.0.104.3:44807)
1027 14:51:15.213673 (+ 564us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:15.213681 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:15.213706 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:51:15.213765 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:15.213771 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:15.213975 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:15.214120 (+ 145us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:15.215527 (+ 1407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.216841 (+ 1314us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:15.217986 (+ 1145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:15.218391 (+ 405us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:15.221928 (+ 3537us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:15.221993 (+ 65us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:15.222009 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:15.222042 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:15.224858 (+ 2816us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.225590 (+ 732us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.225599 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.225601 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.225690 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:15.225959 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:15.225962 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:15.225964 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:15.226235 (+ 271us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:15.226454 (+ 219us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:15.226754 (+ 300us) server_negotiation.cc:300] Negotiation successful
1027 14:51:15.226988 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":433,"thread_start_us":311,"threads_started":1}
W20251027 14:51:15.228713 4556 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='joe-interloper', principal='joe-interloper@KRBTEST.COM'} at 127.0.0.1:59684
W20251027 14:51:15.229892 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4433: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:51:15.230021 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4433
I20251027 14:51:15.242461 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4067
I20251027 14:51:15.243697 4129 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:15.344188 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4067
2025-10-27T14:51:15Z chronyd exiting
[ OK ] SecurityITest.TestUnauthorizedClientKerberosCredentials (4397 ms)
[ RUN ] SecurityITest.TestAuthorizedSuperuser
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4578](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4578](info): set up 2 sockets
Oct 27 14:51:15 dist-test-slave-hxxt krb5kdc[4578](info): commencing operation
krb5kdc: starting...
W20251027 14:51:17.224851 416 mini_kdc.cc:121] Time spent starting KDC: real 1.843s user 0.000s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:17 dist-test-slave-hxxt krb5kdc[4578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:17Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:17Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:17.436206 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43011
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43011
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:17.747674 4594 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:17.747978 4594 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:17.748119 4594 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:17.755965 4594 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:17.756054 4594 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:17.756121 4594 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:17.756183 4594 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:17.756258 4594 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:17.766347 4594 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43011
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:17.767872 4594 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:17.769158 4594 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:17.776644 4600 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:17.776683 4599 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:17.777407 4602 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:17.777837 4594 server_base.cc:1047] running on GCE node
I20251027 14:51:17.778637 4594 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:17.779906 4594 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:17.781121 4594 hybrid_clock.cc:648] HybridClock initialized: now 1761576677781066 us; error 49 us; skew 500 ppm
I20251027 14:51:17.781437 4594 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:17 dist-test-slave-hxxt krb5kdc[4578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576677, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:17.784216 4594 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:17.785740 4594 webserver.cc:492] Webserver started at http://127.0.104.62:44497/ using document root <none> and password file <none>
I20251027 14:51:17.786414 4594 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:17.786496 4594 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:17.786813 4594 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:17.789888 4594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "c6a972158c1d4651a2124e762ae2dd27"
format_stamp: "Formatted at 2025-10-27 14:51:17 on dist-test-slave-hxxt"
server_key: "6ba0a6abfb59d442e42397d2608dc6c2"
server_key_iv: "8c86da04f77da52cec3e29679dcadd36"
server_key_version: "encryptionkey@0"
I20251027 14:51:17.790805 4594 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "c6a972158c1d4651a2124e762ae2dd27"
format_stamp: "Formatted at 2025-10-27 14:51:17 on dist-test-slave-hxxt"
server_key: "6ba0a6abfb59d442e42397d2608dc6c2"
server_key_iv: "8c86da04f77da52cec3e29679dcadd36"
server_key_version: "encryptionkey@0"
I20251027 14:51:17.797101 4594 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20251027 14:51:17.800256 4609 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:17.801539 4594 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:51:17.801687 4594 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "c6a972158c1d4651a2124e762ae2dd27"
format_stamp: "Formatted at 2025-10-27 14:51:17 on dist-test-slave-hxxt"
server_key: "6ba0a6abfb59d442e42397d2608dc6c2"
server_key_iv: "8c86da04f77da52cec3e29679dcadd36"
server_key_version: "encryptionkey@0"
I20251027 14:51:17.801877 4594 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:17.834731 4594 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:17.838732 4594 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:17.838989 4594 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:17.857263 4594 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43011
I20251027 14:51:17.857267 4660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43011 every 8 connection(s)
I20251027 14:51:17.858749 4594 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:17.862783 4661 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:17.866073 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4594
I20251027 14:51:17.866321 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:17.867035 416 external_mini_cluster.cc:1468] Setting key 418a8c81d173fe68ce09bdf84aa7ece8
I20251027 14:51:17.874972 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27: Bootstrap starting.
I20251027 14:51:17.878883 4661 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27: Neither blocks nor log segments found. Creating new log.
Oct 27 14:51:17 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576677, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:17.880116 4661 log.cc:826] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:17.883003 4661 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27: No bootstrap required, opened a new log
I20251027 14:51:17.887271 4664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:17.869135 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44164 (local address 127.0.104.62:43011)
1027 14:51:17.869873 (+ 738us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:17.869884 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:17.869923 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:51:17.871022 (+ 1099us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:17.871045 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:17.871476 (+ 431us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:17.871855 (+ 379us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:17.873819 (+ 1964us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:17.875193 (+ 1374us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:17.876693 (+ 1500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:17.877180 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:17.880834 (+ 3654us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:17.880906 (+ 72us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:17.880924 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:17.880957 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:17.883711 (+ 2754us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:17.884271 (+ 560us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:17.884277 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:17.884284 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:17.884416 (+ 132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:17.884734 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:17.884737 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:17.884740 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:17.885171 (+ 431us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:17.885475 (+ 304us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:17.885873 (+ 398us) server_negotiation.cc:300] Negotiation successful
1027 14:51:17.886253 (+ 380us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":552,"thread_start_us":382,"threads_started":1}
I20251027 14:51:17.888331 4661 raft_consensus.cc:359] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a972158c1d4651a2124e762ae2dd27" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43011 } }
I20251027 14:51:17.888789 4661 raft_consensus.cc:385] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:17.888899 4661 raft_consensus.cc:740] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c6a972158c1d4651a2124e762ae2dd27, State: Initialized, Role: FOLLOWER
I20251027 14:51:17.889572 4661 consensus_queue.cc:260] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a972158c1d4651a2124e762ae2dd27" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43011 } }
I20251027 14:51:17.889806 4661 raft_consensus.cc:399] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:17.889938 4661 raft_consensus.cc:493] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:17.890102 4661 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:17.892220 4661 raft_consensus.cc:515] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a972158c1d4651a2124e762ae2dd27" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43011 } }
I20251027 14:51:17.892719 4661 leader_election.cc:304] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c6a972158c1d4651a2124e762ae2dd27; no voters:
I20251027 14:51:17.893252 4661 leader_election.cc:290] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:17.893361 4666 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:17.893934 4666 raft_consensus.cc:697] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [term 1 LEADER]: Becoming Leader. State: Replica: c6a972158c1d4651a2124e762ae2dd27, State: Running, Role: LEADER
I20251027 14:51:17.894428 4666 consensus_queue.cc:237] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a972158c1d4651a2124e762ae2dd27" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43011 } }
I20251027 14:51:17.895264 4661 sys_catalog.cc:565] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:17.897027 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c6a972158c1d4651a2124e762ae2dd27. Latest consensus state: current_term: 1 leader_uuid: "c6a972158c1d4651a2124e762ae2dd27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a972158c1d4651a2124e762ae2dd27" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43011 } } }
I20251027 14:51:17.897315 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:17.897917 4667 sys_catalog.cc:455] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c6a972158c1d4651a2124e762ae2dd27" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c6a972158c1d4651a2124e762ae2dd27" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43011 } } }
I20251027 14:51:17.898138 4673 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:17.898217 4667 sys_catalog.cc:458] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:17.902935 4673 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:17.911047 4673 catalog_manager.cc:1357] Generated new cluster ID: 8ffdb6737dda45f3bab2cf1c2b53bc84
I20251027 14:51:17.911139 4673 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:17.927955 4673 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:17.929108 4673 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:17.940989 4673 catalog_manager.cc:6022] T 00000000000000000000000000000000 P c6a972158c1d4651a2124e762ae2dd27: Generated new TSK 0
I20251027 14:51:17.941857 4673 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:18.023542 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43011
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:18.334156 4689 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:18.334456 4689 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:18.334585 4689 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:18.342592 4689 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:18.342691 4689 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:18.342873 4689 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:18.353379 4689 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:43011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:18.354918 4689 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:18.356204 4689 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:18.365137 4695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:18.365742 4697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:18.365761 4694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:18.366195 4689 server_base.cc:1047] running on GCE node
I20251027 14:51:18.366780 4689 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:18.367704 4689 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:18.368940 4689 hybrid_clock.cc:648] HybridClock initialized: now 1761576678368860 us; error 37 us; skew 500 ppm
I20251027 14:51:18.369239 4689 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:18 dist-test-slave-hxxt krb5kdc[4578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:18.372254 4689 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:51:18.373879 4689 webserver.cc:492] Webserver started at http://127.0.104.1:36943/ using document root <none> and password file <none>
I20251027 14:51:18.374614 4689 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:18.374704 4689 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:18.375061 4689 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:18.377679 4689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "82a8acbb108d420c82fe530b542df824"
format_stamp: "Formatted at 2025-10-27 14:51:18 on dist-test-slave-hxxt"
server_key: "8c8722c32d1ebdcad97db6bbdec2ea4e"
server_key_iv: "f05cd617bc95c49e4c1543c877e5d4d1"
server_key_version: "encryptionkey@0"
I20251027 14:51:18.378418 4689 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "82a8acbb108d420c82fe530b542df824"
format_stamp: "Formatted at 2025-10-27 14:51:18 on dist-test-slave-hxxt"
server_key: "8c8722c32d1ebdcad97db6bbdec2ea4e"
server_key_iv: "f05cd617bc95c49e4c1543c877e5d4d1"
server_key_version: "encryptionkey@0"
I20251027 14:51:18.383682 4689 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:51:18.387149 4704 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:18.388502 4689 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251027 14:51:18.388659 4689 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "82a8acbb108d420c82fe530b542df824"
format_stamp: "Formatted at 2025-10-27 14:51:18 on dist-test-slave-hxxt"
server_key: "8c8722c32d1ebdcad97db6bbdec2ea4e"
server_key_iv: "f05cd617bc95c49e4c1543c877e5d4d1"
server_key_version: "encryptionkey@0"
I20251027 14:51:18.388865 4689 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:18.410367 4689 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:18.414299 4689 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:18.414553 4689 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:18.415503 4689 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:18.416970 4689 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:18.417076 4689 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:18.417172 4689 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:18.417238 4689 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:18.447119 4689 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:41011
I20251027 14:51:18.447152 4816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:41011 every 8 connection(s)
I20251027 14:51:18.448498 4689 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:18.450508 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4689
I20251027 14:51:18.450677 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:18.451292 416 external_mini_cluster.cc:1468] Setting key a6ad08e9073497e0f3579c91f4e8c064
Oct 27 14:51:18 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:51:18.470561 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:18.451148 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:51177 (local address 127.0.104.62:43011)
1027 14:51:18.451566 (+ 418us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:18.451573 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:18.452798 (+ 1225us) server_negotiation.cc:408] Connection header received
1027 14:51:18.454333 (+ 1535us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:18.454341 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:18.454417 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:18.454532 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:18.456168 (+ 1636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.457385 (+ 1217us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:18.458474 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.458901 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:18.465112 (+ 6211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:18.465145 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:18.465149 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:18.465179 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:18.467576 (+ 2397us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:18.468306 (+ 730us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:18.468311 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:18.468313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:18.468430 (+ 117us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:18.468763 (+ 333us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:18.468769 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:18.468772 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:18.468984 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:18.469123 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:18.469972 (+ 849us) server_negotiation.cc:300] Negotiation successful
1027 14:51:18.470212 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":305,"thread_start_us":206,"threads_started":1}
I20251027 14:51:18.471477 4820 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:18.451765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43011 (local address 127.0.104.1:51177)
1027 14:51:18.452630 (+ 865us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:18.452678 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:18.454051 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:18.454735 (+ 684us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:18.454744 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:18.455314 (+ 570us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:18.455987 (+ 673us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:18.456005 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.457536 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:18.457541 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:18.458329 (+ 788us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:18.458340 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.458597 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:18.462549 (+ 3952us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:18.462574 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:18.464889 (+ 2315us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:18.467728 (+ 2839us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:18.467739 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:18.467760 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:18.468193 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:18.468545 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:18.468549 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:18.468552 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:18.468662 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:18.469140 (+ 478us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:18.469150 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:18.469530 (+ 380us) client_negotiation.cc:769] Sending connection context
1027 14:51:18.469879 (+ 349us) client_negotiation.cc:240] Negotiation successful
1027 14:51:18.470392 (+ 513us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":600,"thread_start_us":407,"threads_started":1}
I20251027 14:51:18.473152 4817 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43011
I20251027 14:51:18.473480 4817 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:18.474249 4817 heartbeater.cc:507] Master 127.0.104.62:43011 requested a full tablet report, sending...
I20251027 14:51:18.476238 4626 ts_manager.cc:194] Registered new tserver with Master: 82a8acbb108d420c82fe530b542df824 (127.0.104.1:41011)
I20251027 14:51:18.477715 4626 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:51177
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:18.523241 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43011
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:18.838279 4825 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:18.838577 4825 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:18.838671 4825 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:18.846748 4825 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:18.846864 4825 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:18.847021 4825 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:18.857540 4825 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:43011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:18.859120 4825 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:18.860477 4825 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:18.869777 4830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:18.870690 4831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:18.872004 4833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:18.872515 4825 server_base.cc:1047] running on GCE node
I20251027 14:51:18.873107 4825 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:18.873983 4825 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:18.875208 4825 hybrid_clock.cc:648] HybridClock initialized: now 1761576678875141 us; error 46 us; skew 500 ppm
I20251027 14:51:18.875484 4825 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:18 dist-test-slave-hxxt krb5kdc[4578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:18.878489 4825 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:51:18.880096 4825 webserver.cc:492] Webserver started at http://127.0.104.2:45991/ using document root <none> and password file <none>
I20251027 14:51:18.880887 4825 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:18.880978 4825 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:18.881333 4825 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:18.883852 4825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "573726a6c86744099072151af4193310"
format_stamp: "Formatted at 2025-10-27 14:51:18 on dist-test-slave-hxxt"
server_key: "9aa1f9b89053660cc1aceadfc6824623"
server_key_iv: "b8e9f1a99e6334fe142d3db5735ee103"
server_key_version: "encryptionkey@0"
I20251027 14:51:18.884635 4825 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "573726a6c86744099072151af4193310"
format_stamp: "Formatted at 2025-10-27 14:51:18 on dist-test-slave-hxxt"
server_key: "9aa1f9b89053660cc1aceadfc6824623"
server_key_iv: "b8e9f1a99e6334fe142d3db5735ee103"
server_key_version: "encryptionkey@0"
I20251027 14:51:18.889812 4825 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20251027 14:51:18.893095 4840 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:18.894426 4825 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:18.894582 4825 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "573726a6c86744099072151af4193310"
format_stamp: "Formatted at 2025-10-27 14:51:18 on dist-test-slave-hxxt"
server_key: "9aa1f9b89053660cc1aceadfc6824623"
server_key_iv: "b8e9f1a99e6334fe142d3db5735ee103"
server_key_version: "encryptionkey@0"
I20251027 14:51:18.894793 4825 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:18.906651 4825 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:18.911058 4825 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:18.911307 4825 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:18.912230 4825 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:18.913700 4825 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:18.913780 4825 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:18.913921 4825 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:18.914000 4825 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:18.941666 4825 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:37477
I20251027 14:51:18.941717 4952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:37477 every 8 connection(s)
I20251027 14:51:18.942977 4825 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:18.950407 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4825
I20251027 14:51:18.950567 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:18.950942 416 external_mini_cluster.cc:1468] Setting key b08bd392ba794c26eb86c0f5eca86c09
Oct 27 14:51:18 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576678, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:18.961213 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:18.945636 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:42739 (local address 127.0.104.62:43011)
1027 14:51:18.945799 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:18.945804 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:18.947022 (+ 1218us) server_negotiation.cc:408] Connection header received
1027 14:51:18.948516 (+ 1494us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:18.948523 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:18.948609 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:18.948702 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:18.950388 (+ 1686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.951311 (+ 923us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:18.952363 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.952855 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:18.955904 (+ 3049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:18.955937 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:18.955941 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:18.955970 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:18.958280 (+ 2310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:18.959124 (+ 844us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:18.959128 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:18.959130 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:18.959189 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:18.959508 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:18.959512 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:18.959514 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:18.959756 (+ 242us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:18.959877 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:18.960632 (+ 755us) server_negotiation.cc:300] Negotiation successful
1027 14:51:18.960899 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:51:18.962118 4955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:18.946160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43011 (local address 127.0.104.2:42739)
1027 14:51:18.946851 (+ 691us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:18.946893 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:18.948209 (+ 1316us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:18.948954 (+ 745us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:18.948962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:18.949510 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:18.950218 (+ 708us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:18.950246 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.951452 (+ 1206us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:18.951456 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:18.952226 (+ 770us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:18.952243 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:18.952517 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:18.953314 (+ 797us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:18.953337 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:18.955687 (+ 2350us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:18.958422 (+ 2735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:18.958432 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:18.958450 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:18.959017 (+ 567us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:18.959297 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:18.959301 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:18.959304 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:18.959410 (+ 106us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:18.959869 (+ 459us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:18.959878 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:18.960183 (+ 305us) client_negotiation.cc:769] Sending connection context
1027 14:51:18.960526 (+ 343us) client_negotiation.cc:240] Negotiation successful
1027 14:51:18.961002 (+ 476us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":472,"thread_start_us":308,"threads_started":1}
I20251027 14:51:18.963528 4953 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43011
I20251027 14:51:18.963841 4953 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:18.964640 4953 heartbeater.cc:507] Master 127.0.104.62:43011 requested a full tablet report, sending...
I20251027 14:51:18.966228 4626 ts_manager.cc:194] Registered new tserver with Master: 573726a6c86744099072151af4193310 (127.0.104.2:37477)
I20251027 14:51:18.967322 4626 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:42739
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:19.023355 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43011
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37651
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:19.340234 4960 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:19.340591 4960 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:19.340725 4960 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:19.348665 4960 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:19.348767 4960 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:19.348937 4960 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:19.358958 4960 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37651
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:43011
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:19.360476 4960 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:19.361730 4960 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:19.370270 4966 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:19.370383 4965 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:19.371019 4968 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:19.371441 4960 server_base.cc:1047] running on GCE node
I20251027 14:51:19.372001 4960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:19.372838 4960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:19.374076 4960 hybrid_clock.cc:648] HybridClock initialized: now 1761576679374001 us; error 41 us; skew 500 ppm
I20251027 14:51:19.374361 4960 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:19.377380 4960 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:51:19.378798 4960 webserver.cc:492] Webserver started at http://127.0.104.3:33369/ using document root <none> and password file <none>
I20251027 14:51:19.379484 4960 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:19.379568 4960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:19.379914 4960 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:19.382465 4960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "e7de6b1dc4d0464bba8dc99086297c83"
format_stamp: "Formatted at 2025-10-27 14:51:19 on dist-test-slave-hxxt"
server_key: "edd94ff7971e63dc873518275f656fd0"
server_key_iv: "8c907f1c858a75befc90943258246c46"
server_key_version: "encryptionkey@0"
I20251027 14:51:19.383203 4960 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "e7de6b1dc4d0464bba8dc99086297c83"
format_stamp: "Formatted at 2025-10-27 14:51:19 on dist-test-slave-hxxt"
server_key: "edd94ff7971e63dc873518275f656fd0"
server_key_iv: "8c907f1c858a75befc90943258246c46"
server_key_version: "encryptionkey@0"
I20251027 14:51:19.388296 4960 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.007s
I20251027 14:51:19.391502 4975 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:19.392903 4960 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20251027 14:51:19.393080 4960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "e7de6b1dc4d0464bba8dc99086297c83"
format_stamp: "Formatted at 2025-10-27 14:51:19 on dist-test-slave-hxxt"
server_key: "edd94ff7971e63dc873518275f656fd0"
server_key_iv: "8c907f1c858a75befc90943258246c46"
server_key_version: "encryptionkey@0"
I20251027 14:51:19.393271 4960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:19.417414 4960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:19.421140 4960 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:19.421396 4960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:19.422323 4960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:19.423771 4960 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:19.423868 4960 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:19.423985 4960 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:19.424055 4960 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:19.453986 4960 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46197
I20251027 14:51:19.454037 5087 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46197 every 8 connection(s)
I20251027 14:51:19.455271 4960 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:19.460422 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 4960
I20251027 14:51:19.460595 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestAuthorizedSuperuser.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:19.461061 416 external_mini_cluster.cc:1468] Setting key c7f365ddbd3449f6ad1f320d754f45fa
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:19.472199 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.457802 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:43181 (local address 127.0.104.62:43011)
1027 14:51:19.457997 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:19.458002 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:19.459092 (+ 1090us) server_negotiation.cc:408] Connection header received
1027 14:51:19.460788 (+ 1696us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:19.460793 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:19.460871 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:19.460967 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:19.462837 (+ 1870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.463849 (+ 1012us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.464932 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.465226 (+ 294us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.467698 (+ 2472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:19.467732 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:19.467736 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:19.467762 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:19.470033 (+ 2271us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.470598 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.470602 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.470604 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.470659 (+ 55us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.470878 (+ 219us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.470881 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.470883 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.471095 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:19.471197 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:19.471780 (+ 583us) server_negotiation.cc:300] Negotiation successful
1027 14:51:19.471973 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:51:19.473026 5090 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.458264 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43011 (local address 127.0.104.3:43181)
1027 14:51:19.458939 (+ 675us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:19.458983 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:19.460480 (+ 1497us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:19.461241 (+ 761us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:19.461253 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:19.461879 (+ 626us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:19.462675 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:19.462701 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.464006 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.464012 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:19.464832 (+ 820us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:19.464843 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.465037 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.465661 (+ 624us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:19.465682 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:19.467546 (+ 1864us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:19.470145 (+ 2599us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:19.470151 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:19.470164 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:19.470519 (+ 355us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:19.470724 (+ 205us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:19.470727 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:19.470729 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:19.470814 (+ 85us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:19.471184 (+ 370us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:19.471200 (+ 16us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:19.471455 (+ 255us) client_negotiation.cc:769] Sending connection context
1027 14:51:19.471713 (+ 258us) client_negotiation.cc:240] Negotiation successful
1027 14:51:19.472041 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":302,"threads_started":1}
I20251027 14:51:19.474308 5088 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43011
I20251027 14:51:19.474650 5088 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:19.475481 5088 heartbeater.cc:507] Master 127.0.104.62:43011 requested a full tablet report, sending...
I20251027 14:51:19.476886 4626 ts_manager.cc:194] Registered new tserver with Master: e7de6b1dc4d0464bba8dc99086297c83 (127.0.104.3:46197)
I20251027 14:51:19.477623 4626 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:43181
I20251027 14:51:19.480494 4817 heartbeater.cc:499] Master 127.0.104.62:43011 was elected leader, sending a full tablet report...
I20251027 14:51:19.487888 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:51:19.531944 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.518915 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52454 (local address 127.0.104.1:41011)
1027 14:51:19.519471 (+ 556us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:19.519479 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:19.519567 (+ 88us) server_negotiation.cc:408] Connection header received
1027 14:51:19.519732 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:19.519740 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:19.519964 (+ 224us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:19.520113 (+ 149us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:19.521579 (+ 1466us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.522762 (+ 1183us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.523884 (+ 1122us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.524241 (+ 357us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.527317 (+ 3076us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:19.527388 (+ 71us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:19.527404 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:19.527435 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:19.529938 (+ 2503us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.530505 (+ 567us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.530511 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.530514 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.530586 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.530781 (+ 195us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.530784 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.530786 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.531058 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:19.531296 (+ 238us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:19.531461 (+ 165us) server_negotiation.cc:300] Negotiation successful
1027 14:51:19.531665 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":399,"thread_start_us":304,"threads_started":1}
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:19.547219 4819 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.535989 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.0.104.62:43011)
1027 14:51:19.536178 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:19.536182 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:19.536234 (+ 52us) server_negotiation.cc:408] Connection header received
1027 14:51:19.536392 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:19.536399 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:19.536467 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:19.536544 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:19.537796 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.538688 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.539710 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.540031 (+ 321us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.543026 (+ 2995us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:19.543063 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:19.543066 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:19.543098 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:19.545395 (+ 2297us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.545900 (+ 505us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.545904 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.545906 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.545963 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.546149 (+ 186us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.546152 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.546154 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.546401 (+ 247us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:19.546578 (+ 177us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:19.546753 (+ 175us) server_negotiation.cc:300] Negotiation successful
1027 14:51:19.546921 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20251027 14:51:19.547745 4626 server_base.cc:1129] Unauthorized access attempt to method kudu.master.MasterService.TSHeartbeat from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44180
I20251027 14:51:19.548734 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4689
I20251027 14:51:19.559607 5095 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.549642 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52458 (local address 127.0.104.1:41011)
1027 14:51:19.549815 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:19.549821 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:19.549929 (+ 108us) server_negotiation.cc:408] Connection header received
1027 14:51:19.550091 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:19.550095 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:19.550153 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:19.550229 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:19.551465 (+ 1236us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.552292 (+ 827us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.553375 (+ 1083us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.553837 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.555166 (+ 1329us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:19.555201 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:19.555204 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:19.555233 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:19.557499 (+ 2266us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.558051 (+ 552us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.558055 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.558058 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.558130 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.558387 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.558392 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.558395 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.558599 (+ 204us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:19.558814 (+ 215us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:19.559078 (+ 264us) server_negotiation.cc:300] Negotiation successful
1027 14:51:19.559323 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:51:19.560207 4812 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:19.642190 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4689
I20251027 14:51:19.655005 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4825
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:51:19.669713 5100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.655987 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.104.2:37477)
1027 14:51:19.656515 (+ 528us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:19.656523 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:19.656549 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:51:19.656612 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:19.656619 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:19.656817 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:19.656995 (+ 178us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:19.658339 (+ 1344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.659480 (+ 1141us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.660630 (+ 1150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.661042 (+ 412us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.664267 (+ 3225us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:19.664330 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:19.664345 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:19.664402 (+ 57us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:19.667150 (+ 2748us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.667774 (+ 624us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.667782 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.667785 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.667855 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.668153 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.668158 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.668162 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.668570 (+ 408us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:19.668851 (+ 281us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:19.669132 (+ 281us) server_negotiation.cc:300] Negotiation successful
1027 14:51:19.669417 (+ 285us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":368,"thread_start_us":263,"threads_started":1}
I20251027 14:51:19.670965 4948 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:19.751515 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4825
I20251027 14:51:19.763906 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4960
Oct 27 14:51:19 dist-test-slave-hxxt krb5kdc[4578](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576679, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:51:19.778766 5102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:19.764870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56350 (local address 127.0.104.3:46197)
1027 14:51:19.765454 (+ 584us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:19.765461 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:19.765506 (+ 45us) server_negotiation.cc:408] Connection header received
1027 14:51:19.765568 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:19.765574 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:19.765776 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:19.765957 (+ 181us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:19.767407 (+ 1450us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.768730 (+ 1323us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:19.769881 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:19.770317 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:19.773623 (+ 3306us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:19.773684 (+ 61us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:19.773699 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:19.773729 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:19.776329 (+ 2600us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.777019 (+ 690us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.777026 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.777029 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.777118 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:19.777378 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:19.777383 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:19.777387 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:19.777645 (+ 258us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:19.777915 (+ 270us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:19.778216 (+ 301us) server_negotiation.cc:300] Negotiation successful
1027 14:51:19.778468 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":445,"thread_start_us":321,"threads_started":1}
I20251027 14:51:19.780019 5083 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:19.861775 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4960
I20251027 14:51:19.875109 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 4594
I20251027 14:51:19.876256 4656 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:19.976253 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 4594
2025-10-27T14:51:19Z chronyd exiting
[ OK ] SecurityITest.TestAuthorizedSuperuser (4632 ms)
[ RUN ] SecurityITest.TestDisableWebUI
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:20 dist-test-slave-hxxt krb5kdc[5107](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:20 dist-test-slave-hxxt krb5kdc[5107](info): set up 2 sockets
Oct 27 14:51:20 dist-test-slave-hxxt krb5kdc[5107](info): commencing operation
krb5kdc: starting...
W20251027 14:51:21.856946 416 mini_kdc.cc:121] Time spent starting KDC: real 1.843s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:21 dist-test-slave-hxxt krb5kdc[5107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:22.075950 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46231
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:46231
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:22.391412 5123 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:22.391773 5123 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:22.391903 5123 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:22.401475 5123 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:22.401613 5123 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:22.401722 5123 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:22.401793 5123 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:22.401872 5123 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:22.413092 5123 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:46231
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46231
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:22.414721 5123 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:22.416052 5123 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:22.423704 5128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:22.423866 5129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:22.423977 5131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:22.424911 5123 server_base.cc:1047] running on GCE node
I20251027 14:51:22.425631 5123 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:22.427003 5123 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:22.428211 5123 hybrid_clock.cc:648] HybridClock initialized: now 1761576682428147 us; error 41 us; skew 500 ppm
I20251027 14:51:22.428539 5123 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:22 dist-test-slave-hxxt krb5kdc[5107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576682, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:22.431433 5123 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:22.432629 5123 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:22.432757 5123 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:22.433079 5123 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:22.435525 5123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "7ac7afd6b9874d7b92c942990c674440"
format_stamp: "Formatted at 2025-10-27 14:51:22 on dist-test-slave-hxxt"
server_key: "b02b754c9c0d96c704bf580d49a8e171"
server_key_iv: "f1ac68e48edd8ded41a64fe5b6ea4d34"
server_key_version: "encryptionkey@0"
I20251027 14:51:22.436245 5123 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "7ac7afd6b9874d7b92c942990c674440"
format_stamp: "Formatted at 2025-10-27 14:51:22 on dist-test-slave-hxxt"
server_key: "b02b754c9c0d96c704bf580d49a8e171"
server_key_iv: "f1ac68e48edd8ded41a64fe5b6ea4d34"
server_key_version: "encryptionkey@0"
I20251027 14:51:22.441318 5123 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:51:22.444635 5137 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:22.445957 5123 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:51:22.446113 5123 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "7ac7afd6b9874d7b92c942990c674440"
format_stamp: "Formatted at 2025-10-27 14:51:22 on dist-test-slave-hxxt"
server_key: "b02b754c9c0d96c704bf580d49a8e171"
server_key_iv: "f1ac68e48edd8ded41a64fe5b6ea4d34"
server_key_version: "encryptionkey@0"
I20251027 14:51:22.446283 5123 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:22.466416 5123 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:22.470103 5123 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:22.470355 5123 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:22.488386 5123 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46231
I20251027 14:51:22.488416 5188 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46231 every 8 connection(s)
I20251027 14:51:22.489352 5123 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:22.493095 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5123
I20251027 14:51:22.493283 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:22.493458 5189 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:22.493857 416 external_mini_cluster.cc:1468] Setting key 9a015f66b627bced2e9572276382cb5b
Oct 27 14:51:22 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:22.506524 5189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440: Bootstrap starting.
I20251027 14:51:22.510649 5189 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:22.511919 5189 log.cc:826] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:22.514549 5192 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:22.495667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40318 (local address 127.0.104.62:46231)
1027 14:51:22.496513 (+ 846us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:22.496529 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:22.496590 (+ 61us) server_negotiation.cc:408] Connection header received
1027 14:51:22.497967 (+ 1377us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:22.497996 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:22.498544 (+ 548us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:22.499081 (+ 537us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:22.500436 (+ 1355us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:22.502333 (+ 1897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:22.503513 (+ 1180us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:22.504099 (+ 586us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:22.506878 (+ 2779us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:22.506989 (+ 111us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:22.507011 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:22.507059 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:22.510581 (+ 3522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:22.511197 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:22.511206 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:22.511214 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:22.511326 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:22.511586 (+ 260us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:22.511592 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:22.511595 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:22.512205 (+ 610us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:22.512548 (+ 343us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:22.512850 (+ 302us) server_negotiation.cc:300] Negotiation successful
1027 14:51:22.513269 (+ 419us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":585,"thread_start_us":367,"threads_started":1}
I20251027 14:51:22.515348 5189 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440: No bootstrap required, opened a new log
I20251027 14:51:22.520584 5189 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac7afd6b9874d7b92c942990c674440" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46231 } }
I20251027 14:51:22.520928 5189 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:22.521035 5189 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7ac7afd6b9874d7b92c942990c674440, State: Initialized, Role: FOLLOWER
I20251027 14:51:22.521621 5189 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac7afd6b9874d7b92c942990c674440" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46231 } }
I20251027 14:51:22.521864 5189 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:22.522024 5189 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:22.522195 5189 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:22.524490 5189 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac7afd6b9874d7b92c942990c674440" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46231 } }
I20251027 14:51:22.524974 5189 leader_election.cc:304] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7ac7afd6b9874d7b92c942990c674440; no voters:
I20251027 14:51:22.525496 5189 leader_election.cc:290] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:22.525648 5194 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:22.526688 5194 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [term 1 LEADER]: Becoming Leader. State: Replica: 7ac7afd6b9874d7b92c942990c674440, State: Running, Role: LEADER
I20251027 14:51:22.527187 5194 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac7afd6b9874d7b92c942990c674440" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46231 } }
I20251027 14:51:22.527555 5189 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:22.529789 5196 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7ac7afd6b9874d7b92c942990c674440. Latest consensus state: current_term: 1 leader_uuid: "7ac7afd6b9874d7b92c942990c674440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac7afd6b9874d7b92c942990c674440" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46231 } } }
I20251027 14:51:22.530064 5196 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:22.530524 5195 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7ac7afd6b9874d7b92c942990c674440" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7ac7afd6b9874d7b92c942990c674440" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46231 } } }
I20251027 14:51:22.530797 5195 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:22.531124 5201 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:22.536731 5201 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:22.544756 5201 catalog_manager.cc:1357] Generated new cluster ID: b1d3a6cb04a64f47bd527298f5f3cae4
I20251027 14:51:22.544875 5201 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:22.575352 5201 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:22.577198 5201 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:22.587920 5201 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440: Generated new TSK 0
I20251027 14:51:22.589074 5201 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:22.679800 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46231
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:23.002952 5217 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:23.003244 5217 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:23.003376 5217 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:23.011322 5217 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:23.011423 5217 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:23.011624 5217 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:23.021921 5217 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:46231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:23.023634 5217 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:23.024993 5217 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:23.033918 5223 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:23.033900 5222 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:23.034667 5225 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:23.035115 5217 server_base.cc:1047] running on GCE node
I20251027 14:51:23.035627 5217 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:23.036541 5217 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:23.037750 5217 hybrid_clock.cc:648] HybridClock initialized: now 1761576683037683 us; error 51 us; skew 500 ppm
I20251027 14:51:23.038038 5217 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:23 dist-test-slave-hxxt krb5kdc[5107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:23.041170 5217 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:51:23.042299 5217 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:23.042416 5217 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:23.042743 5217 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:23.045351 5217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "3081bb6870fb4eb8b469368744b091f1"
format_stamp: "Formatted at 2025-10-27 14:51:23 on dist-test-slave-hxxt"
server_key: "d8c9f3d2b1acded66fd09ed0c2f1ebfe"
server_key_iv: "8f66de3e9212b780068b696385f6d3fd"
server_key_version: "encryptionkey@0"
I20251027 14:51:23.046109 5217 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "3081bb6870fb4eb8b469368744b091f1"
format_stamp: "Formatted at 2025-10-27 14:51:23 on dist-test-slave-hxxt"
server_key: "d8c9f3d2b1acded66fd09ed0c2f1ebfe"
server_key_iv: "8f66de3e9212b780068b696385f6d3fd"
server_key_version: "encryptionkey@0"
I20251027 14:51:23.051426 5217 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:51:23.054965 5231 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:23.056282 5217 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:51:23.056488 5217 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "3081bb6870fb4eb8b469368744b091f1"
format_stamp: "Formatted at 2025-10-27 14:51:23 on dist-test-slave-hxxt"
server_key: "d8c9f3d2b1acded66fd09ed0c2f1ebfe"
server_key_iv: "8f66de3e9212b780068b696385f6d3fd"
server_key_version: "encryptionkey@0"
I20251027 14:51:23.056691 5217 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:23.084200 5217 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:23.088506 5217 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:23.088784 5217 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:23.089718 5217 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:23.091255 5217 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:23.091328 5217 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:23.091423 5217 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:23.091478 5217 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:23.122466 5217 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:36495
I20251027 14:51:23.122488 5343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:36495 every 8 connection(s)
I20251027 14:51:23.123317 5217 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:23.130174 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5217
I20251027 14:51:23.130389 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:23.131073 416 external_mini_cluster.cc:1468] Setting key f2e3d9f89b86f4fc45fab4fae8dbc1d4
Oct 27 14:51:23 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:23.142704 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:23.125813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:39655 (local address 127.0.104.62:46231)
1027 14:51:23.126230 (+ 417us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:23.126235 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:23.127246 (+ 1011us) server_negotiation.cc:408] Connection header received
1027 14:51:23.128881 (+ 1635us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:23.128889 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:23.128979 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:23.129082 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:23.131220 (+ 2138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.132396 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:23.133821 (+ 1425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.134186 (+ 365us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:23.137396 (+ 3210us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:23.137426 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:23.137430 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:23.137462 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:23.139881 (+ 2419us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:23.140597 (+ 716us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:23.140603 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:23.140605 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:23.140683 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:23.141006 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:23.141010 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:23.141012 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:23.141238 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:23.141376 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:23.142125 (+ 749us) server_negotiation.cc:300] Negotiation successful
1027 14:51:23.142367 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":218,"threads_started":1}
I20251027 14:51:23.143642 5347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:23.126344 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46231 (local address 127.0.104.1:39655)
1027 14:51:23.127074 (+ 730us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:23.127113 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:23.128585 (+ 1472us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:23.129337 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:23.129347 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:23.130025 (+ 678us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:23.130991 (+ 966us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:23.131017 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.132566 (+ 1549us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:23.132571 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:23.133653 (+ 1082us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:23.133669 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.133954 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:23.134711 (+ 757us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:23.134744 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:23.137184 (+ 2440us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:23.140020 (+ 2836us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:23.140030 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:23.140050 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:23.140483 (+ 433us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:23.140780 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:23.140786 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:23.140789 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:23.140899 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:23.141354 (+ 455us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:23.141363 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:23.141704 (+ 341us) client_negotiation.cc:769] Sending connection context
1027 14:51:23.142003 (+ 299us) client_negotiation.cc:240] Negotiation successful
1027 14:51:23.142477 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":304,"threads_started":1}
I20251027 14:51:23.145218 5344 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46231
I20251027 14:51:23.145637 5344 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:23.146574 5344 heartbeater.cc:507] Master 127.0.104.62:46231 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:51:23.149216 5154 ts_manager.cc:194] Registered new tserver with Master: 3081bb6870fb4eb8b469368744b091f1 (127.0.104.1:36495)
I20251027 14:51:23.150748 5154 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:39655
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:23.207608 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46231
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:23.518396 5352 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:23.518689 5352 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:23.518822 5352 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:23.526870 5352 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:23.526961 5352 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:23.527133 5352 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:23.537458 5352 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:46231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:23.538944 5352 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:23.540207 5352 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:23.549403 5360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:23.549427 5357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:23.549561 5358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:23.550208 5352 server_base.cc:1047] running on GCE node
I20251027 14:51:23.550779 5352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:23.551680 5352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:23.552896 5352 hybrid_clock.cc:648] HybridClock initialized: now 1761576683552822 us; error 44 us; skew 500 ppm
I20251027 14:51:23.553157 5352 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:23 dist-test-slave-hxxt krb5kdc[5107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:23.556090 5352 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:51:23.557194 5352 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:23.557308 5352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:23.557613 5352 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:23.560174 5352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "774b35851c2e48b0935b803d5387422c"
format_stamp: "Formatted at 2025-10-27 14:51:23 on dist-test-slave-hxxt"
server_key: "4009f0148f91e8b711952f29e34761aa"
server_key_iv: "41cd9d6f06439f8c28c6f375cd6a2276"
server_key_version: "encryptionkey@0"
I20251027 14:51:23.560945 5352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "774b35851c2e48b0935b803d5387422c"
format_stamp: "Formatted at 2025-10-27 14:51:23 on dist-test-slave-hxxt"
server_key: "4009f0148f91e8b711952f29e34761aa"
server_key_iv: "41cd9d6f06439f8c28c6f375cd6a2276"
server_key_version: "encryptionkey@0"
I20251027 14:51:23.566124 5352 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:51:23.569505 5366 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:23.570758 5352 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20251027 14:51:23.570914 5352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "774b35851c2e48b0935b803d5387422c"
format_stamp: "Formatted at 2025-10-27 14:51:23 on dist-test-slave-hxxt"
server_key: "4009f0148f91e8b711952f29e34761aa"
server_key_iv: "41cd9d6f06439f8c28c6f375cd6a2276"
server_key_version: "encryptionkey@0"
I20251027 14:51:23.571094 5352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:23.591671 5352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:23.600189 5352 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:23.600507 5352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:23.601447 5352 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:23.602985 5352 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:23.603063 5352 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:23.603178 5352 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:23.603230 5352 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:23.632104 5352 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41957
I20251027 14:51:23.632140 5478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41957 every 8 connection(s)
I20251027 14:51:23.632983 5352 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:23.634972 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5352
I20251027 14:51:23.635146 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:23.635637 416 external_mini_cluster.cc:1468] Setting key 6a23da3ea5bbc29d3bbf0503c96d4b80
Oct 27 14:51:23 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576683, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:23.650996 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:23.635497 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:46341 (local address 127.0.104.62:46231)
1027 14:51:23.635685 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:23.635692 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:23.637115 (+ 1423us) server_negotiation.cc:408] Connection header received
1027 14:51:23.638706 (+ 1591us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:23.638712 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:23.638795 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:23.638888 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:23.640623 (+ 1735us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.641453 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:23.642553 (+ 1100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.643000 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:23.645664 (+ 2664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:23.645701 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:23.645705 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:23.645734 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:23.648098 (+ 2364us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:23.648834 (+ 736us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:23.648839 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:23.648841 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:23.648909 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:23.649219 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:23.649223 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:23.649225 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:23.649454 (+ 229us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:23.649579 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:23.650345 (+ 766us) server_negotiation.cc:300] Negotiation successful
1027 14:51:23.650690 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":82}
I20251027 14:51:23.651911 5481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:23.636166 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46231 (local address 127.0.104.2:46341)
1027 14:51:23.636966 (+ 800us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:23.637005 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:23.638427 (+ 1422us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:23.639155 (+ 728us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:23.639165 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:23.639720 (+ 555us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:23.640452 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:23.640472 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.641623 (+ 1151us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:23.641627 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:23.642417 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:23.642428 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:23.642688 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:23.643425 (+ 737us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:23.643449 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:23.645470 (+ 2021us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:23.648241 (+ 2771us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:23.648251 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:23.648270 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:23.648713 (+ 443us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:23.649004 (+ 291us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:23.649007 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:23.649010 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:23.649115 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:23.649572 (+ 457us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:23.649580 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:23.649908 (+ 328us) client_negotiation.cc:769] Sending connection context
1027 14:51:23.650226 (+ 318us) client_negotiation.cc:240] Negotiation successful
1027 14:51:23.650696 (+ 470us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":543,"thread_start_us":356,"threads_started":1}
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:51:23.653337 5479 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46231
I20251027 14:51:23.653666 5479 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:23.654376 5479 heartbeater.cc:507] Master 127.0.104.62:46231 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:51:23.655881 5154 ts_manager.cc:194] Registered new tserver with Master: 774b35851c2e48b0935b803d5387422c (127.0.104.2:41957)
I20251027 14:51:23.656697 5154 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:46341
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:23.709403 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46231
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38863
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--webserver_enabled=0 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:24.025684 5486 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:24.025956 5486 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:24.026059 5486 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:24.034121 5486 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:24.034219 5486 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:24.034373 5486 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:24.044625 5486 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38863
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--webserver_enabled=false
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:46231
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:24.046111 5486 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:24.047401 5486 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:24.056149 5492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:24.056332 5491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:24.056708 5494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:24.057365 5486 server_base.cc:1047] running on GCE node
I20251027 14:51:24.057933 5486 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:24.058840 5486 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:24.060048 5486 hybrid_clock.cc:648] HybridClock initialized: now 1761576684059983 us; error 38 us; skew 500 ppm
I20251027 14:51:24.060328 5486 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:24 dist-test-slave-hxxt krb5kdc[5107](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:24.063544 5486 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:51:24.064765 5486 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:24.064875 5486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:24.065168 5486 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:24.067760 5486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "2cd6c05191094bc3b2a7dc5c51b36d22"
format_stamp: "Formatted at 2025-10-27 14:51:24 on dist-test-slave-hxxt"
server_key: "0238378a827e8555d1dae5c97756d03c"
server_key_iv: "ada3728ac6c9ac51c0b1f8d5cc87ff7e"
server_key_version: "encryptionkey@0"
I20251027 14:51:24.068495 5486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "2cd6c05191094bc3b2a7dc5c51b36d22"
format_stamp: "Formatted at 2025-10-27 14:51:24 on dist-test-slave-hxxt"
server_key: "0238378a827e8555d1dae5c97756d03c"
server_key_iv: "ada3728ac6c9ac51c0b1f8d5cc87ff7e"
server_key_version: "encryptionkey@0"
I20251027 14:51:24.073707 5486 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:51:24.077020 5500 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:24.078263 5486 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:24.078409 5486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "2cd6c05191094bc3b2a7dc5c51b36d22"
format_stamp: "Formatted at 2025-10-27 14:51:24 on dist-test-slave-hxxt"
server_key: "0238378a827e8555d1dae5c97756d03c"
server_key_iv: "ada3728ac6c9ac51c0b1f8d5cc87ff7e"
server_key_version: "encryptionkey@0"
I20251027 14:51:24.078588 5486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:24.090157 5486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:24.093739 5486 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:24.093993 5486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:24.094848 5486 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:24.096292 5486 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:24.096362 5486 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:24.096516 5486 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:24.096573 5486 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:24.126914 5486 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:42839
I20251027 14:51:24.127002 5612 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:42839 every 8 connection(s)
I20251027 14:51:24.127753 5486 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:24.129065 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5486
I20251027 14:51:24.129252 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:24.129730 416 external_mini_cluster.cc:1468] Setting key 28121da0a854af7ffbf0cfe35d7cfa16
Oct 27 14:51:24 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576684, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:24.145480 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.130415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:43537 (local address 127.0.104.62:46231)
1027 14:51:24.130599 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.130604 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.131975 (+ 1371us) server_negotiation.cc:408] Connection header received
1027 14:51:24.133620 (+ 1645us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.133625 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.133699 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.133776 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:24.135379 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.136224 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.137286 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.137593 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.140255 (+ 2662us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:24.140294 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:24.140298 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:24.140328 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:24.142574 (+ 2246us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.143191 (+ 617us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.143196 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.143198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.143273 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.143592 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.143597 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.143600 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.143876 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:24.144007 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.144930 (+ 923us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.145167 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:51:24.146355 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.130976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46231 (local address 127.0.104.3:43537)
1027 14:51:24.131799 (+ 823us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:24.131860 (+ 61us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:24.133383 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:24.133986 (+ 603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:24.133994 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:24.134575 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:24.135241 (+ 666us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.135259 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.136362 (+ 1103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.136367 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:24.137183 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.137194 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.137398 (+ 204us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.138047 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:24.138072 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:24.140062 (+ 1990us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:24.142702 (+ 2640us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:24.142709 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:24.142722 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:24.143085 (+ 363us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:24.143368 (+ 283us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:24.143373 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:24.143376 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:24.143488 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:24.144011 (+ 523us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:24.144019 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:24.144431 (+ 412us) client_negotiation.cc:769] Sending connection context
1027 14:51:24.144814 (+ 383us) client_negotiation.cc:240] Negotiation successful
1027 14:51:24.145265 (+ 451us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":364,"threads_started":1}
I20251027 14:51:24.147617 5613 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46231
I20251027 14:51:24.147929 5613 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:24.148648 5613 heartbeater.cc:507] Master 127.0.104.62:46231 requested a full tablet report, sending...
I20251027 14:51:24.149997 5154 ts_manager.cc:194] Registered new tserver with Master: 2cd6c05191094bc3b2a7dc5c51b36d22 (127.0.104.3:42839)
I20251027 14:51:24.150791 5154 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:43537
I20251027 14:51:24.154197 5344 heartbeater.cc:499] Master 127.0.104.62:46231 was elected leader, sending a full tablet report...
I20251027 14:51:24.156783 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:51:24.171293 5346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.160873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40324 (local address 127.0.104.62:46231)
1027 14:51:24.161016 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.161020 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.161466 (+ 446us) server_negotiation.cc:408] Connection header received
1027 14:51:24.161613 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.161617 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.161677 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.161751 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:24.163191 (+ 1440us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.164058 (+ 867us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.165152 (+ 1094us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.165485 (+ 333us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.167047 (+ 1562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:24.167078 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:24.167082 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:24.167110 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:24.169521 (+ 2411us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.170010 (+ 489us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.170013 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.170016 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.170073 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.170263 (+ 190us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.170266 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.170268 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.170507 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:24.170710 (+ 203us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.170917 (+ 207us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.171077 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251027 14:51:24.174747 5154 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40324:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:51:24.179260 5154 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:51:24.199992 5627 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.190717 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:36495 (local address 127.0.0.1:41284)
1027 14:51:24.192209 (+ 1492us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:24.192250 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:24.192465 (+ 215us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:24.193659 (+ 1194us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:24.193674 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:24.193700 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:24.195108 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.195118 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.197059 (+ 1941us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.197064 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:24.199066 (+ 2002us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.199081 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.199274 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.199346 (+ 72us) client_negotiation.cc:769] Sending connection context
1027 14:51:24.199490 (+ 144us) client_negotiation.cc:240] Negotiation successful
1027 14:51:24.199669 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1222,"thread_start_us":2635,"threads_started":1}
I20251027 14:51:24.200253 5626 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.191011 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:42839 (local address 127.0.0.1:44338)
1027 14:51:24.193550 (+ 2539us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:24.193564 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:24.193675 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:24.194528 (+ 853us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:24.194533 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:24.194561 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:24.195378 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.195390 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.197646 (+ 2256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.197653 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:24.199436 (+ 1783us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.199450 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.199688 (+ 238us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.199714 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:51:24.199783 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:51:24.199900 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2459,"thread_start_us":933,"threads_started":1}
I20251027 14:51:24.201303 5630 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.194398 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:41957 (local address 127.0.0.1:54216)
1027 14:51:24.194987 (+ 589us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:24.195003 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:24.195115 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:24.196058 (+ 943us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:24.196062 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:24.196087 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:24.196712 (+ 625us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.196724 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.198717 (+ 1993us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.198723 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:24.200665 (+ 1942us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.200679 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.200834 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.200857 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:51:24.200920 (+ 63us) client_negotiation.cc:240] Negotiation successful
1027 14:51:24.201023 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":319,"threads_started":1}
I20251027 14:51:24.201472 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.191640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44338 (local address 127.0.104.3:42839)
1027 14:51:24.193360 (+ 1720us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.193368 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.193787 (+ 419us) server_negotiation.cc:408] Connection header received
1027 14:51:24.193897 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.193905 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.194137 (+ 232us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.194337 (+ 200us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:24.195575 (+ 1238us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.197213 (+ 1638us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.199639 (+ 2426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.200779 (+ 1140us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.200937 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.201058 (+ 121us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.201214 (+ 156us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1566,"thread_start_us":1452,"threads_started":1}
I20251027 14:51:24.202131 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.190786 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41284 (local address 127.0.104.1:36495)
1027 14:51:24.193036 (+ 2250us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.193047 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.193076 (+ 29us) server_negotiation.cc:408] Connection header received
1027 14:51:24.193185 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.193195 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.193451 (+ 256us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.193656 (+ 205us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:24.195279 (+ 1623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.196880 (+ 1601us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.200441 (+ 3561us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.201463 (+ 1022us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.201614 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.201731 (+ 117us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.201899 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2077,"thread_start_us":1967,"threads_started":1}
I20251027 14:51:24.203289 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.193917 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54216 (local address 127.0.104.2:41957)
1027 14:51:24.195536 (+ 1619us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.195545 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.195570 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:51:24.195657 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.195665 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.195879 (+ 214us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.196072 (+ 193us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:24.196865 (+ 793us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.198445 (+ 1580us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.200803 (+ 2358us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.201910 (+ 1107us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.202031 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.202143 (+ 112us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.203039 (+ 896us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1470,"thread_start_us":1375,"threads_started":1}
I20251027 14:51:24.206673 5414 tablet_service.cc:1505] Processing CreateTablet for tablet 93d632476cc04edfaa620a9bc433857f (DEFAULT_TABLE table=test-table [id=2aa6ca8602454b39940f38c6611ece50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:51:24.206542 5279 tablet_service.cc:1505] Processing CreateTablet for tablet 93d632476cc04edfaa620a9bc433857f (DEFAULT_TABLE table=test-table [id=2aa6ca8602454b39940f38c6611ece50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:51:24.206542 5548 tablet_service.cc:1505] Processing CreateTablet for tablet 93d632476cc04edfaa620a9bc433857f (DEFAULT_TABLE table=test-table [id=2aa6ca8602454b39940f38c6611ece50]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:51:24.207937 5279 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d632476cc04edfaa620a9bc433857f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:24.207955 5414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d632476cc04edfaa620a9bc433857f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:24.207968 5548 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93d632476cc04edfaa620a9bc433857f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:24.218298 5631 tablet_bootstrap.cc:492] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Bootstrap starting.
I20251027 14:51:24.221313 5632 tablet_bootstrap.cc:492] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Bootstrap starting.
I20251027 14:51:24.222930 5633 tablet_bootstrap.cc:492] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Bootstrap starting.
I20251027 14:51:24.223176 5631 tablet_bootstrap.cc:654] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:24.224658 5631 log.cc:826] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:24.225262 5632 tablet_bootstrap.cc:654] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:24.225847 5633 tablet_bootstrap.cc:654] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:24.226719 5632 log.cc:826] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:24.226809 5633 log.cc:826] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:24.229842 5633 tablet_bootstrap.cc:492] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: No bootstrap required, opened a new log
I20251027 14:51:24.230180 5633 ts_tablet_manager.cc:1403] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Time spent bootstrapping tablet: real 0.007s user 0.007s sys 0.000s
I20251027 14:51:24.231037 5632 tablet_bootstrap.cc:492] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: No bootstrap required, opened a new log
I20251027 14:51:24.231232 5631 tablet_bootstrap.cc:492] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: No bootstrap required, opened a new log
I20251027 14:51:24.231285 5632 ts_tablet_manager.cc:1403] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20251027 14:51:24.231529 5631 ts_tablet_manager.cc:1403] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.006s
I20251027 14:51:24.236310 5632 raft_consensus.cc:359] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.236692 5632 raft_consensus.cc:385] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:24.236821 5632 raft_consensus.cc:740] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3081bb6870fb4eb8b469368744b091f1, State: Initialized, Role: FOLLOWER
I20251027 14:51:24.237242 5633 raft_consensus.cc:359] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.237430 5632 consensus_queue.cc:260] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.237591 5633 raft_consensus.cc:385] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:24.237739 5633 raft_consensus.cc:740] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2cd6c05191094bc3b2a7dc5c51b36d22, State: Initialized, Role: FOLLOWER
I20251027 14:51:24.238510 5633 consensus_queue.cc:260] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.238587 5631 raft_consensus.cc:359] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.238953 5631 raft_consensus.cc:385] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:24.239079 5632 ts_tablet_manager.cc:1434] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251027 14:51:24.239120 5631 raft_consensus.cc:740] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 774b35851c2e48b0935b803d5387422c, State: Initialized, Role: FOLLOWER
I20251027 14:51:24.239878 5631 consensus_queue.cc:260] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.240065 5613 heartbeater.cc:499] Master 127.0.104.62:46231 was elected leader, sending a full tablet report...
I20251027 14:51:24.240700 5633 ts_tablet_manager.cc:1434] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20251027 14:51:24.242456 5631 ts_tablet_manager.cc:1434] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Time spent starting tablet: real 0.011s user 0.010s sys 0.002s
I20251027 14:51:24.242688 5479 heartbeater.cc:499] Master 127.0.104.62:46231 was elected leader, sending a full tablet report...
I20251027 14:51:24.374763 5638 raft_consensus.cc:493] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:51:24.375042 5638 raft_consensus.cc:515] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
W20251027 14:51:24.375376 5345 tablet.cc:2378] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:51:24.376852 5638 leader_election.cc:290] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3081bb6870fb4eb8b469368744b091f1 (127.0.104.1:36495), 774b35851c2e48b0935b803d5387422c (127.0.104.2:41957)
W20251027 14:51:24.379132 5614 tablet.cc:2378] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:51:24.382033 5615 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.377057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:36495 (local address 127.0.104.3:58045)
1027 14:51:24.377217 (+ 160us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:24.377236 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:24.377373 (+ 137us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:24.377852 (+ 479us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:24.377858 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:24.377885 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:24.378307 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.378318 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.379715 (+ 1397us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.379721 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:24.381372 (+ 1651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.381387 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.381541 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.381566 (+ 25us) client_negotiation.cc:769] Sending connection context
1027 14:51:24.381628 (+ 62us) client_negotiation.cc:240] Negotiation successful
1027 14:51:24.381750 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":46}
I20251027 14:51:24.382501 5640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.377390 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:41957 (local address 127.0.104.3:53125)
1027 14:51:24.377904 (+ 514us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:24.377923 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:24.378034 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:24.378308 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:24.378312 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:24.378337 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:24.378862 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.378870 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.380238 (+ 1368us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.380244 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:24.381863 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:24.381875 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.382051 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.382075 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:51:24.382144 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:51:24.382250 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":266,"threads_started":1}
I20251027 14:51:24.382900 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.377211 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:58045 (local address 127.0.104.1:36495)
1027 14:51:24.377390 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.377397 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.377420 (+ 23us) server_negotiation.cc:408] Connection header received
1027 14:51:24.377495 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.377501 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.377601 (+ 100us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.377697 (+ 96us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:24.378445 (+ 748us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.379562 (+ 1117us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.381558 (+ 1996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.382433 (+ 875us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.382484 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.382574 (+ 90us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.382694 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:51:24.383507 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.377589 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:53125 (local address 127.0.104.2:41957)
1027 14:51:24.377754 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.377760 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.378009 (+ 249us) server_negotiation.cc:408] Connection header received
1027 14:51:24.378150 (+ 141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.378154 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.378209 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.378276 (+ 67us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:24.378995 (+ 719us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.380056 (+ 1061us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.382030 (+ 1974us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.383098 (+ 1068us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.383145 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.383213 (+ 68us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.383321 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:51:24.383896 5299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d632476cc04edfaa620a9bc433857f" candidate_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3081bb6870fb4eb8b469368744b091f1" is_pre_election: true
I20251027 14:51:24.384449 5299 raft_consensus.cc:2468] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd6c05191094bc3b2a7dc5c51b36d22 in term 0.
W20251027 14:51:24.384616 5480 tablet.cc:2378] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:51:24.384490 5434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d632476cc04edfaa620a9bc433857f" candidate_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774b35851c2e48b0935b803d5387422c" is_pre_election: true
I20251027 14:51:24.384964 5434 raft_consensus.cc:2468] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2cd6c05191094bc3b2a7dc5c51b36d22 in term 0.
I20251027 14:51:24.385186 5504 leader_election.cc:304] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd6c05191094bc3b2a7dc5c51b36d22, 3081bb6870fb4eb8b469368744b091f1; no voters:
I20251027 14:51:24.385576 5638 raft_consensus.cc:2804] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:51:24.385715 5638 raft_consensus.cc:493] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:51:24.385819 5638 raft_consensus.cc:3060] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:24.387959 5638 raft_consensus.cc:515] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.388538 5638 leader_election.cc:290] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [CANDIDATE]: Term 1 election: Requested vote from peers 3081bb6870fb4eb8b469368744b091f1 (127.0.104.1:36495), 774b35851c2e48b0935b803d5387422c (127.0.104.2:41957)
I20251027 14:51:24.388876 5299 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d632476cc04edfaa620a9bc433857f" candidate_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3081bb6870fb4eb8b469368744b091f1"
I20251027 14:51:24.388917 5434 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "93d632476cc04edfaa620a9bc433857f" candidate_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "774b35851c2e48b0935b803d5387422c"
I20251027 14:51:24.389040 5299 raft_consensus.cc:3060] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:24.389065 5434 raft_consensus.cc:3060] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:24.391222 5299 raft_consensus.cc:2468] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd6c05191094bc3b2a7dc5c51b36d22 in term 1.
I20251027 14:51:24.391247 5434 raft_consensus.cc:2468] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2cd6c05191094bc3b2a7dc5c51b36d22 in term 1.
I20251027 14:51:24.391594 5504 leader_election.cc:304] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2cd6c05191094bc3b2a7dc5c51b36d22, 774b35851c2e48b0935b803d5387422c; no voters:
I20251027 14:51:24.391860 5638 raft_consensus.cc:2804] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:24.392252 5638 raft_consensus.cc:697] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 1 LEADER]: Becoming Leader. State: Replica: 2cd6c05191094bc3b2a7dc5c51b36d22, State: Running, Role: LEADER
I20251027 14:51:24.392700 5638 consensus_queue.cc:237] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } }
I20251027 14:51:24.397173 5152 catalog_manager.cc:5649] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2cd6c05191094bc3b2a7dc5c51b36d22 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "2cd6c05191094bc3b2a7dc5c51b36d22" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 42839 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 } health_report { overall_health: UNKNOWN } } }
I20251027 14:51:24.442927 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.438525 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44340 (local address 127.0.104.3:42839)
1027 14:51:24.438692 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.438697 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.438714 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:51:24.438873 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.438878 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.438962 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.439079 (+ 117us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:51:24.439518 (+ 439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.440361 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.441468 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.441784 (+ 316us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.441940 (+ 156us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:51:24.442438 (+ 498us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:51:24.442555 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.442590 (+ 35us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.442667 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:51:24.450402 5434 raft_consensus.cc:1275] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 1 FOLLOWER]: Refusing update from remote peer 2cd6c05191094bc3b2a7dc5c51b36d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:51:24.450398 5299 raft_consensus.cc:1275] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 1 FOLLOWER]: Refusing update from remote peer 2cd6c05191094bc3b2a7dc5c51b36d22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:51:24.451279 5641 consensus_queue.cc:1048] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3081bb6870fb4eb8b469368744b091f1" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 36495 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:51:24.451526 5638 consensus_queue.cc:1048] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "774b35851c2e48b0935b803d5387422c" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41957 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:51:24.461647 5646 mvcc.cc:204] Tried to move back new op lower bound from 7215418099499343872 to 7215418099276558336. Current Snapshot: MvccSnapshot[applied={T|T < 7215418099499343872}]
I20251027 14:51:24.478214 5152 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:51:24.478475 5152 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40324:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:51:24.481848 5152 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 7ac7afd6b9874d7b92c942990c674440: Sending DeleteTablet for 3 replicas of tablet 93d632476cc04edfaa620a9bc433857f
I20251027 14:51:24.482913 5279 tablet_service.cc:1552] Processing DeleteTablet for tablet 93d632476cc04edfaa620a9bc433857f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:24 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:41284
I20251027 14:51:24.483037 5548 tablet_service.cc:1552] Processing DeleteTablet for tablet 93d632476cc04edfaa620a9bc433857f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:24 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:44338
I20251027 14:51:24.483876 5414 tablet_service.cc:1552] Processing DeleteTablet for tablet 93d632476cc04edfaa620a9bc433857f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:24 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:54216
I20251027 14:51:24.484925 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5217
I20251027 14:51:24.492906 5655 tablet_replica.cc:333] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: stopping tablet replica
I20251027 14:51:24.493377 5655 raft_consensus.cc:2243] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:51:24.493901 5655 raft_consensus.cc:2272] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:24.494742 5653 tablet_replica.cc:333] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: stopping tablet replica
I20251027 14:51:24.495234 5653 raft_consensus.cc:2243] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:51:24.495772 5653 raft_consensus.cc:2272] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:24.498147 5655 ts_tablet_manager.cc:1916] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:24.498152 5653 ts_tablet_manager.cc:1916] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:24.504988 5654 tablet_replica.cc:333] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: stopping tablet replica
I20251027 14:51:24.505532 5654 raft_consensus.cc:2243] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:51:24.505606 5653 ts_tablet_manager.cc:1929] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:51:24.505750 5653 log.cc:1199] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/93d632476cc04edfaa620a9bc433857f
I20251027 14:51:24.506148 5653 ts_tablet_manager.cc:1950] T 93d632476cc04edfaa620a9bc433857f P 3081bb6870fb4eb8b469368744b091f1: Deleting consensus metadata
I20251027 14:51:24.506354 5654 raft_consensus.cc:2272] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:24.507946 5141 catalog_manager.cc:4985] TS 3081bb6870fb4eb8b469368744b091f1 (127.0.104.1:36495): tablet 93d632476cc04edfaa620a9bc433857f (table test-table [id=2aa6ca8602454b39940f38c6611ece50]) successfully deleted
I20251027 14:51:24.510593 5655 ts_tablet_manager.cc:1929] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:51:24.510859 5655 log.cc:1199] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/93d632476cc04edfaa620a9bc433857f
I20251027 14:51:24.511463 5655 ts_tablet_manager.cc:1950] T 93d632476cc04edfaa620a9bc433857f P 774b35851c2e48b0935b803d5387422c: Deleting consensus metadata
I20251027 14:51:24.511515 5654 ts_tablet_manager.cc:1916] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:24.513504 5141 catalog_manager.cc:4985] TS 774b35851c2e48b0935b803d5387422c (127.0.104.2:41957): tablet 93d632476cc04edfaa620a9bc433857f (table test-table [id=2aa6ca8602454b39940f38c6611ece50]) successfully deleted
I20251027 14:51:24.519063 5654 ts_tablet_manager.cc:1929] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:51:24.519311 5654 log.cc:1199] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableWebUI.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/93d632476cc04edfaa620a9bc433857f
I20251027 14:51:24.519919 5654 ts_tablet_manager.cc:1950] T 93d632476cc04edfaa620a9bc433857f P 2cd6c05191094bc3b2a7dc5c51b36d22: Deleting consensus metadata
Oct 27 14:51:24 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:51:24.521824 5140 catalog_manager.cc:4985] TS 2cd6c05191094bc3b2a7dc5c51b36d22 (127.0.104.3:42839): tablet 93d632476cc04edfaa620a9bc433857f (table test-table [id=2aa6ca8602454b39940f38c6611ece50]) successfully deleted
I20251027 14:51:24.527818 5625 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.486271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41298 (local address 127.0.104.1:36495)
1027 14:51:24.486445 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.486452 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.508991 (+ 22539us) server_negotiation.cc:408] Connection header received
1027 14:51:24.509346 (+ 355us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.509352 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.509436 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.509539 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:24.514279 (+ 4740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.515538 (+ 1259us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.517195 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.517651 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.522443 (+ 4792us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:24.522502 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:24.522517 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:24.522552 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:24.525389 (+ 2837us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.526034 (+ 645us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.526042 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.526044 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.526122 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.526410 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.526415 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.526417 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.526704 (+ 287us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:24.526983 (+ 279us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.527270 (+ 287us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.527524 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:51:24.528625 5339 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:24.621505 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5217
I20251027 14:51:24.636202 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5352
Oct 27 14:51:24 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:51:24.650382 5629 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.637073 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54226 (local address 127.0.104.2:41957)
1027 14:51:24.637226 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.637231 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.637251 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:51:24.637446 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.637457 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.637519 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.637604 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:24.638934 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.639857 (+ 923us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.641135 (+ 1278us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.641631 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.644790 (+ 3159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:24.644833 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:24.644848 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:24.644879 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:24.647783 (+ 2904us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.648497 (+ 714us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.648505 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.648507 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.648585 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.648895 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.648901 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.648903 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.649188 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:24.649488 (+ 300us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.649809 (+ 321us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.650049 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:51:24.651084 5474 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:24.737937 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5352
I20251027 14:51:24.751770 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5486
Oct 27 14:51:24 dist-test-slave-hxxt krb5kdc[5107](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576681, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:51:24.767832 5628 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:24.752472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44346 (local address 127.0.104.3:42839)
1027 14:51:24.752637 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:24.752641 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:24.752657 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:51:24.752867 (+ 210us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:24.752872 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:24.752932 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:24.753016 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:24.754853 (+ 1837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.755790 (+ 937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:24.757320 (+ 1530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:24.757692 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:24.762111 (+ 4419us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:24.762150 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:24.762164 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:24.762212 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:24.765303 (+ 3091us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.766077 (+ 774us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.766083 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.766086 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.766180 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:24.766503 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:24.766506 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:24.766508 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:24.766799 (+ 291us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:24.767029 (+ 230us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:24.767324 (+ 295us) server_negotiation.cc:300] Negotiation successful
1027 14:51:24.767502 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251027 14:51:24.768559 5608 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:24.885679 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5486
I20251027 14:51:24.900688 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5123
I20251027 14:51:24.901823 5184 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:25.006376 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5123
2025-10-27T14:51:25Z chronyd exiting
[ OK ] SecurityITest.TestDisableWebUI (5030 ms)
[ RUN ] SecurityITest.TestDisableAuthenticationEncryption
2025-10-27T14:51:25Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:25Z Disabled control of system clock
I20251027 14:51:25.067643 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38377
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38377
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251027 14:51:25.377589 5664 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:25.377884 5664 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:25.377969 5664 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:25.385809 5664 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:25.385913 5664 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:25.385960 5664 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:25.386008 5664 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:25.386065 5664 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:25.395983 5664 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38377
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38377
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:25.397544 5664 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:25.399297 5664 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:25.406774 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:25.406899 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:25.407279 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:25.407756 5664 server_base.cc:1047] running on GCE node
I20251027 14:51:25.408550 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:25.409868 5664 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:25.411128 5664 hybrid_clock.cc:648] HybridClock initialized: now 1761576685411030 us; error 43 us; skew 500 ppm
I20251027 14:51:25.411520 5664 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:25.413700 5664 webserver.cc:492] Webserver started at http://127.0.104.62:46663/ using document root <none> and password file <none>
I20251027 14:51:25.414405 5664 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:25.414515 5664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:25.414829 5664 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:25.417392 5664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "b570e6d6f0884810a31fc7a5c37be2aa"
format_stamp: "Formatted at 2025-10-27 14:51:25 on dist-test-slave-hxxt"
server_key: "16307d3c5648048a82d9fb2075331c4d"
server_key_iv: "762a61264a0503195fde1a33a4c4f976"
server_key_version: "encryptionkey@0"
I20251027 14:51:25.418125 5664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "b570e6d6f0884810a31fc7a5c37be2aa"
format_stamp: "Formatted at 2025-10-27 14:51:25 on dist-test-slave-hxxt"
server_key: "16307d3c5648048a82d9fb2075331c4d"
server_key_iv: "762a61264a0503195fde1a33a4c4f976"
server_key_version: "encryptionkey@0"
I20251027 14:51:25.423485 5664 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:51:25.426786 5678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:25.428059 5664 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:51:25.428197 5664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "b570e6d6f0884810a31fc7a5c37be2aa"
format_stamp: "Formatted at 2025-10-27 14:51:25 on dist-test-slave-hxxt"
server_key: "16307d3c5648048a82d9fb2075331c4d"
server_key_iv: "762a61264a0503195fde1a33a4c4f976"
server_key_version: "encryptionkey@0"
I20251027 14:51:25.428429 5664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:25.434370 5664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:25.435101 5664 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:25.435340 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:25.451749 5664 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38377
I20251027 14:51:25.451778 5729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38377 every 8 connection(s)
I20251027 14:51:25.453244 5664 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:25.457306 5730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:25.463481 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5664
I20251027 14:51:25.463675 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:25.464223 416 external_mini_cluster.cc:1468] Setting key 3c1a57167c622ea0a8f3d10a5f193667
I20251027 14:51:25.467459 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa: Bootstrap starting.
I20251027 14:51:25.470989 5730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:25.471819 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:25.466142 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57666 (local address 127.0.104.62:38377)
1027 14:51:25.467094 (+ 952us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:25.467112 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:25.467151 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:51:25.468296 (+ 1145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:25.468318 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:25.468798 (+ 480us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:25.469193 (+ 395us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:25.470284 (+ 1091us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:25.470291 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:25.470303 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:25.470315 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:25.470373 (+ 58us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:25.470387 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:25.470533 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:25.470682 (+ 149us) server_negotiation.cc:300] Negotiation successful
1027 14:51:25.470912 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":541,"thread_start_us":370,"threads_started":1}
I20251027 14:51:25.472108 5730 log.cc:826] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:25.475365 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa: No bootstrap required, opened a new log
I20251027 14:51:25.480265 5730 raft_consensus.cc:359] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38377 } }
I20251027 14:51:25.480597 5730 raft_consensus.cc:385] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:25.480676 5730 raft_consensus.cc:740] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b570e6d6f0884810a31fc7a5c37be2aa, State: Initialized, Role: FOLLOWER
I20251027 14:51:25.481144 5730 consensus_queue.cc:260] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38377 } }
I20251027 14:51:25.481307 5730 raft_consensus.cc:399] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:25.481393 5730 raft_consensus.cc:493] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:25.481515 5730 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:25.483794 5730 raft_consensus.cc:515] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38377 } }
I20251027 14:51:25.484231 5730 leader_election.cc:304] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b570e6d6f0884810a31fc7a5c37be2aa; no voters:
I20251027 14:51:25.484783 5730 leader_election.cc:290] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:25.484900 5735 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:25.485462 5735 raft_consensus.cc:697] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [term 1 LEADER]: Becoming Leader. State: Replica: b570e6d6f0884810a31fc7a5c37be2aa, State: Running, Role: LEADER
I20251027 14:51:25.485949 5735 consensus_queue.cc:237] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38377 } }
I20251027 14:51:25.486582 5730 sys_catalog.cc:565] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:25.488574 5737 sys_catalog.cc:455] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [sys.catalog]: SysCatalogTable state changed. Reason: New leader b570e6d6f0884810a31fc7a5c37be2aa. Latest consensus state: current_term: 1 leader_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38377 } } }
I20251027 14:51:25.488539 5736 sys_catalog.cc:455] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b570e6d6f0884810a31fc7a5c37be2aa" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38377 } } }
I20251027 14:51:25.489303 5736 sys_catalog.cc:458] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:25.489303 5737 sys_catalog.cc:458] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:25.490749 5741 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:25.496074 5741 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:25.504241 5741 catalog_manager.cc:1357] Generated new cluster ID: 3117de485415428eabcb8780f2f51373
I20251027 14:51:25.504360 5741 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:25.529451 5741 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:25.529897 5741 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:25.542948 5741 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa: Generated new TSK 0
I20251027 14:51:25.544051 5741 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:51:25.552512 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38377
--builtin_ntp_servers=127.0.104.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
W20251027 14:51:25.873157 5754 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:25.873462 5754 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:25.873555 5754 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:25.881676 5754 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:25.881773 5754 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:25.881904 5754 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:25.891549 5754 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:25.893191 5754 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:25.894973 5754 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:25.903872 5760 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:25.903872 5759 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:25.904600 5762 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:25.905032 5754 server_base.cc:1047] running on GCE node
I20251027 14:51:25.905537 5754 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:25.906361 5754 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:25.907563 5754 hybrid_clock.cc:648] HybridClock initialized: now 1761576685907512 us; error 34 us; skew 500 ppm
I20251027 14:51:25.907835 5754 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:25.909976 5754 webserver.cc:492] Webserver started at http://127.0.104.1:33961/ using document root <none> and password file <none>
I20251027 14:51:25.910666 5754 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:25.910782 5754 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:25.911094 5754 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:25.913628 5754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "0482873b7ca743a1ae24cf1a6fad151e"
format_stamp: "Formatted at 2025-10-27 14:51:25 on dist-test-slave-hxxt"
server_key: "251ce844d048016c05fb4e9d13df6493"
server_key_iv: "4a8f432bbd8c5fb79a45e468f2718482"
server_key_version: "encryptionkey@0"
I20251027 14:51:25.914356 5754 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "0482873b7ca743a1ae24cf1a6fad151e"
format_stamp: "Formatted at 2025-10-27 14:51:25 on dist-test-slave-hxxt"
server_key: "251ce844d048016c05fb4e9d13df6493"
server_key_iv: "4a8f432bbd8c5fb79a45e468f2718482"
server_key_version: "encryptionkey@0"
I20251027 14:51:25.919327 5754 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:51:25.922544 5768 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:25.923805 5754 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:51:25.923943 5754 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "0482873b7ca743a1ae24cf1a6fad151e"
format_stamp: "Formatted at 2025-10-27 14:51:25 on dist-test-slave-hxxt"
server_key: "251ce844d048016c05fb4e9d13df6493"
server_key_iv: "4a8f432bbd8c5fb79a45e468f2718482"
server_key_version: "encryptionkey@0"
I20251027 14:51:25.924114 5754 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:25.930130 5754 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:25.930851 5754 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:25.931097 5754 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:25.931929 5754 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:25.933458 5754 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:25.933530 5754 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:25.933667 5754 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:25.933719 5754 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:25.962411 5754 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39103
I20251027 14:51:25.962472 5880 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39103 every 8 connection(s)
I20251027 14:51:25.963759 5754 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:25.970552 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5754
I20251027 14:51:25.970796 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:25.971537 416 external_mini_cluster.cc:1468] Setting key 0f36c26efa622b462fd164b739f54eb9
I20251027 14:51:25.971926 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:25.966417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:46813 (local address 127.0.104.62:38377)
1027 14:51:25.966674 (+ 257us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:25.966679 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:25.968019 (+ 1340us) server_negotiation.cc:408] Connection header received
1027 14:51:25.969551 (+ 1532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:25.969559 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:25.969628 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:25.969717 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:25.971174 (+ 1457us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:25.971182 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:25.971187 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:25.971205 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:25.971244 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:25.971261 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:25.971379 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:25.971674 (+ 295us) server_negotiation.cc:300] Negotiation successful
1027 14:51:25.971730 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":134}
I20251027 14:51:25.973237 5883 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:25.966853 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38377 (local address 127.0.104.1:46813)
1027 14:51:25.967845 (+ 992us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:25.967881 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:25.969247 (+ 1366us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:25.969954 (+ 707us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:25.969962 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:25.970060 (+ 98us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:25.970811 (+ 751us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:25.970814 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:25.970852 (+ 38us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:25.970969 (+ 117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:25.971379 (+ 410us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:25.971389 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:25.971397 (+ 8us) client_negotiation.cc:769] Sending connection context
1027 14:51:25.971744 (+ 347us) client_negotiation.cc:240] Negotiation successful
1027 14:51:25.972001 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":530,"thread_start_us":321,"threads_started":1}
I20251027 14:51:25.974576 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38377
--builtin_ntp_servers=127.0.104.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251027 14:51:25.974917 5881 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38377
I20251027 14:51:25.975227 5881 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:25.975795 5881 heartbeater.cc:507] Master 127.0.104.62:38377 requested a full tablet report, sending...
I20251027 14:51:25.977988 5695 ts_manager.cc:194] Registered new tserver with Master: 0482873b7ca743a1ae24cf1a6fad151e (127.0.104.1:39103)
W20251027 14:51:26.285070 5884 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:26.285351 5884 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:26.285472 5884 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:26.293411 5884 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:26.293495 5884 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:26.293649 5884 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:26.302923 5884 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:26.304518 5884 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:26.306185 5884 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:26.315111 5890 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:26.315140 5889 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:26.315660 5892 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:26.315991 5884 server_base.cc:1047] running on GCE node
I20251027 14:51:26.316617 5884 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:26.317469 5884 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:26.318688 5884 hybrid_clock.cc:648] HybridClock initialized: now 1761576686318625 us; error 43 us; skew 500 ppm
I20251027 14:51:26.318971 5884 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:26.321152 5884 webserver.cc:492] Webserver started at http://127.0.104.2:41141/ using document root <none> and password file <none>
I20251027 14:51:26.321895 5884 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:26.322011 5884 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:26.322324 5884 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:26.324913 5884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "573c0f9118c048caab6a512fda2a1b9f"
format_stamp: "Formatted at 2025-10-27 14:51:26 on dist-test-slave-hxxt"
server_key: "124452858b4063522ab6df16c505f563"
server_key_iv: "946435d5f9ae8a59ebe5cd40f430cfff"
server_key_version: "encryptionkey@0"
I20251027 14:51:26.325667 5884 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "573c0f9118c048caab6a512fda2a1b9f"
format_stamp: "Formatted at 2025-10-27 14:51:26 on dist-test-slave-hxxt"
server_key: "124452858b4063522ab6df16c505f563"
server_key_iv: "946435d5f9ae8a59ebe5cd40f430cfff"
server_key_version: "encryptionkey@0"
I20251027 14:51:26.330830 5884 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20251027 14:51:26.334151 5898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:26.335376 5884 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:51:26.335532 5884 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "573c0f9118c048caab6a512fda2a1b9f"
format_stamp: "Formatted at 2025-10-27 14:51:26 on dist-test-slave-hxxt"
server_key: "124452858b4063522ab6df16c505f563"
server_key_iv: "946435d5f9ae8a59ebe5cd40f430cfff"
server_key_version: "encryptionkey@0"
I20251027 14:51:26.335721 5884 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:26.341862 5884 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:26.342561 5884 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:26.342818 5884 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:26.343653 5884 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:26.345151 5884 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:26.345229 5884 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:26.345341 5884 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:26.345392 5884 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:26.374749 5884 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:41445
I20251027 14:51:26.374783 6010 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:41445 every 8 connection(s)
I20251027 14:51:26.376077 5884 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:26.381918 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 5884
I20251027 14:51:26.382110 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:26.382556 416 external_mini_cluster.cc:1468] Setting key 386e78afa16a4978009cf53cef2fdf49
I20251027 14:51:26.383912 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.378653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:48199 (local address 127.0.104.62:38377)
1027 14:51:26.378796 (+ 143us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:26.378800 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:26.380198 (+ 1398us) server_negotiation.cc:408] Connection header received
1027 14:51:26.381568 (+ 1370us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:26.381573 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:26.381624 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:26.381708 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:26.383150 (+ 1442us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:26.383156 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:26.383160 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:26.383177 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:26.383216 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:26.383232 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:26.383357 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:26.383593 (+ 236us) server_negotiation.cc:300] Negotiation successful
1027 14:51:26.383655 (+ 62us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":47}
I20251027 14:51:26.385020 6013 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.379144 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38377 (local address 127.0.104.2:48199)
1027 14:51:26.380034 (+ 890us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:26.380073 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:26.381321 (+ 1248us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:26.381969 (+ 648us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:26.381981 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:26.382110 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:26.382826 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:26.382829 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:26.382865 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:26.382949 (+ 84us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:26.383332 (+ 383us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:26.383340 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:26.383347 (+ 7us) client_negotiation.cc:769] Sending connection context
1027 14:51:26.383677 (+ 330us) client_negotiation.cc:240] Negotiation successful
1027 14:51:26.383863 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":471,"thread_start_us":289,"threads_started":1}
I20251027 14:51:26.385521 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38377
--builtin_ntp_servers=127.0.104.20:43465
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled
--rpc_encryption=disabled with env {}
I20251027 14:51:26.386250 6011 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38377
I20251027 14:51:26.386602 6011 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:26.387230 6011 heartbeater.cc:507] Master 127.0.104.62:38377 requested a full tablet report, sending...
I20251027 14:51:26.388684 5695 ts_manager.cc:194] Registered new tserver with Master: 573c0f9118c048caab6a512fda2a1b9f (127.0.104.2:41445)
W20251027 14:51:26.705487 6014 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:26.705803 6014 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:26.705924 6014 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:26.716190 6014 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:26.716284 6014 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:26.716481 6014 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:26.726047 6014 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43465
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:38377
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:26.727607 6014 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:26.729429 6014 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:26.738369 6019 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:26.738333 6020 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:26.738929 6022 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:26.739401 6014 server_base.cc:1047] running on GCE node
I20251027 14:51:26.739935 6014 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:26.740831 6014 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:26.742041 6014 hybrid_clock.cc:648] HybridClock initialized: now 1761576686741961 us; error 50 us; skew 500 ppm
I20251027 14:51:26.742342 6014 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:26.744323 6014 webserver.cc:492] Webserver started at http://127.0.104.3:45577/ using document root <none> and password file <none>
I20251027 14:51:26.745031 6014 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:26.745126 6014 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:26.745401 6014 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:26.747828 6014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5"
format_stamp: "Formatted at 2025-10-27 14:51:26 on dist-test-slave-hxxt"
server_key: "17225892a5a7bbe60f66a70cca8403d1"
server_key_iv: "56c652ff06f9bbd7347cf8ded9289c13"
server_key_version: "encryptionkey@0"
I20251027 14:51:26.748560 6014 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5"
format_stamp: "Formatted at 2025-10-27 14:51:26 on dist-test-slave-hxxt"
server_key: "17225892a5a7bbe60f66a70cca8403d1"
server_key_iv: "56c652ff06f9bbd7347cf8ded9289c13"
server_key_version: "encryptionkey@0"
I20251027 14:51:26.753558 6014 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:51:26.756778 6028 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:26.758011 6014 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251027 14:51:26.758157 6014 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5"
format_stamp: "Formatted at 2025-10-27 14:51:26 on dist-test-slave-hxxt"
server_key: "17225892a5a7bbe60f66a70cca8403d1"
server_key_iv: "56c652ff06f9bbd7347cf8ded9289c13"
server_key_version: "encryptionkey@0"
I20251027 14:51:26.758335 6014 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:26.764617 6014 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:26.765332 6014 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:26.765592 6014 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:26.766429 6014 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:26.767915 6014 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:26.767990 6014 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:26.768105 6014 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:26.768157 6014 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:26.799054 6014 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34843
I20251027 14:51:26.799072 6140 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34843 every 8 connection(s)
I20251027 14:51:26.800937 6014 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:26.803764 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6014
I20251027 14:51:26.803952 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:26.804452 416 external_mini_cluster.cc:1468] Setting key 3d0872b88f8d91cc254c8d26e0ae29fb
I20251027 14:51:26.809506 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.804551 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:56437 (local address 127.0.104.62:38377)
1027 14:51:26.804749 (+ 198us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:26.804755 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:26.805924 (+ 1169us) server_negotiation.cc:408] Connection header received
1027 14:51:26.807414 (+ 1490us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:26.807420 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:26.807495 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:26.807590 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:26.808845 (+ 1255us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:26.808851 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:26.808856 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:26.808873 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:26.808903 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:26.808914 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:26.809013 (+ 99us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:26.809218 (+ 205us) server_negotiation.cc:300] Negotiation successful
1027 14:51:26.809278 (+ 60us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251027 14:51:26.810457 6143 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.804697 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38377 (local address 127.0.104.3:56437)
1027 14:51:26.805795 (+ 1098us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:26.805831 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:26.807166 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:26.807838 (+ 672us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:26.807847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:26.807938 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:26.808602 (+ 664us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:26.808604 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:26.808631 (+ 27us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:26.808693 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:26.808994 (+ 301us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:26.809003 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:26.809011 (+ 8us) client_negotiation.cc:769] Sending connection context
1027 14:51:26.809292 (+ 281us) client_negotiation.cc:240] Negotiation successful
1027 14:51:26.809542 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":354,"threads_started":1}
I20251027 14:51:26.811578 6141 heartbeater.cc:344] Connected to a master server at 127.0.104.62:38377
I20251027 14:51:26.811851 6141 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:26.812353 6141 heartbeater.cc:507] Master 127.0.104.62:38377 requested a full tablet report, sending...
I20251027 14:51:26.813700 5695 ts_manager.cc:194] Registered new tserver with Master: 6dab693e2e7d4bb1b7a673d3a1a028b5 (127.0.104.3:34843)
I20251027 14:51:26.819715 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:51:26.826424 5733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.824182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57670 (local address 127.0.104.62:38377)
1027 14:51:26.824343 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:26.824347 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:26.824807 (+ 460us) server_negotiation.cc:408] Connection header received
1027 14:51:26.824946 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:26.824950 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:26.825000 (+ 50us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:26.825081 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:26.825972 (+ 891us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:26.825976 (+ 4us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:26.825979 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:26.825989 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:26.826013 (+ 24us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:26.826022 (+ 9us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:26.826106 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:26.826168 (+ 62us) server_negotiation.cc:300] Negotiation successful
1027 14:51:26.826210 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:51:26.830469 5695 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:57670:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:51:26.835690 5695 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:51:26.852742 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.847112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:41445 (local address 127.0.0.1:50380)
1027 14:51:26.848299 (+ 1187us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:26.848336 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:26.848511 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:26.850237 (+ 1726us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:26.850245 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:26.850374 (+ 129us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:26.851170 (+ 796us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:26.851174 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:26.851241 (+ 67us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:26.851342 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:26.851893 (+ 551us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:26.851905 (+ 12us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:26.851914 (+ 9us) client_negotiation.cc:769] Sending connection context
1027 14:51:26.852114 (+ 200us) client_negotiation.cc:240] Negotiation successful
1027 14:51:26.852212 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":957,"thread_start_us":874,"threads_started":1}
I20251027 14:51:26.853581 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.847315 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50380 (local address 127.0.104.2:41445)
1027 14:51:26.847925 (+ 610us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:26.847936 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:26.848465 (+ 529us) server_negotiation.cc:408] Connection header received
1027 14:51:26.848639 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:26.848649 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:26.848891 (+ 242us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:26.849070 (+ 179us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:26.851542 (+ 2472us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:26.851555 (+ 13us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:26.851574 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:26.851594 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:26.851656 (+ 62us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:26.851676 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:26.851840 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:26.853117 (+ 1277us) server_negotiation.cc:300] Negotiation successful
1027 14:51:26.853219 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":437,"thread_start_us":323,"threads_started":1}
I20251027 14:51:26.860654 6153 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.848761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35646)
1027 14:51:26.853971 (+ 5210us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:26.853991 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:26.854079 (+ 88us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:26.858311 (+ 4232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:26.858315 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:26.858332 (+ 17us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:26.859025 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:26.859028 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:26.859051 (+ 23us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:26.859098 (+ 47us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:26.860190 (+ 1092us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:26.860195 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:26.860200 (+ 5us) client_negotiation.cc:769] Sending connection context
1027 14:51:26.860330 (+ 130us) client_negotiation.cc:240] Negotiation successful
1027 14:51:26.860400 (+ 70us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5117,"thread_start_us":8370,"threads_started":1}
I20251027 14:51:26.860746 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.849241 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35646 (local address 127.0.104.1:39103)
1027 14:51:26.857709 (+ 8468us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:26.857720 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:26.857755 (+ 35us) server_negotiation.cc:408] Connection header received
1027 14:51:26.857827 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:26.857837 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:26.858101 (+ 264us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:26.858648 (+ 547us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:26.859978 (+ 1330us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:26.859987 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:26.860000 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:26.860013 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:26.860056 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:26.860073 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:26.860227 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:26.860347 (+ 120us) server_negotiation.cc:300] Negotiation successful
1027 14:51:26.860456 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8270,"thread_start_us":8473,"threads_started":1}
I20251027 14:51:26.862039 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.848809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56616 (local address 127.0.104.3:34843)
1027 14:51:26.858343 (+ 9534us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:26.858351 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:26.858386 (+ 35us) server_negotiation.cc:408] Connection header received
1027 14:51:26.858452 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:26.858460 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:26.858658 (+ 198us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:26.859230 (+ 572us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:26.861380 (+ 2150us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:26.861387 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:26.861396 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:26.861408 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:26.861438 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:26.861449 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:26.861610 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:26.861663 (+ 53us) server_negotiation.cc:300] Negotiation successful
1027 14:51:26.861746 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9365,"thread_start_us":9244,"threads_started":1}
I20251027 14:51:26.862076 6156 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:26.848771 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34843 (local address 127.0.0.1:56616)
1027 14:51:26.856832 (+ 8061us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:26.856851 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:26.856955 (+ 104us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:26.858866 (+ 1911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:26.858872 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:26.858893 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:26.859501 (+ 608us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:26.859504 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:26.859517 (+ 13us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:26.859545 (+ 28us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:26.861544 (+ 1999us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:26.861549 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:26.861554 (+ 5us) client_negotiation.cc:769] Sending connection context
1027 14:51:26.861665 (+ 111us) client_negotiation.cc:240] Negotiation successful
1027 14:51:26.861710 (+ 45us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7933,"mutex_wait_us":1,"thread_start_us":7828,"threads_started":1}
I20251027 14:51:26.862144 5946 tablet_service.cc:1505] Processing CreateTablet for tablet 51341b68073b445ba7ab13244f440cee (DEFAULT_TABLE table=test-table [id=aef93b0f0aa44e17aab163ade03125fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:51:26.863919 5946 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51341b68073b445ba7ab13244f440cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:26.865072 5816 tablet_service.cc:1505] Processing CreateTablet for tablet 51341b68073b445ba7ab13244f440cee (DEFAULT_TABLE table=test-table [id=aef93b0f0aa44e17aab163ade03125fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:51:26.866094 6076 tablet_service.cc:1505] Processing CreateTablet for tablet 51341b68073b445ba7ab13244f440cee (DEFAULT_TABLE table=test-table [id=aef93b0f0aa44e17aab163ade03125fc]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:51:26.866494 5816 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51341b68073b445ba7ab13244f440cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:26.867163 6076 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 51341b68073b445ba7ab13244f440cee. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:26.877377 6159 tablet_bootstrap.cc:492] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Bootstrap starting.
I20251027 14:51:26.880577 6159 tablet_bootstrap.cc:654] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:26.881131 6161 tablet_bootstrap.cc:492] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: Bootstrap starting.
I20251027 14:51:26.881956 6160 tablet_bootstrap.cc:492] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Bootstrap starting.
I20251027 14:51:26.882015 6159 log.cc:826] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:26.885217 6159 tablet_bootstrap.cc:492] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: No bootstrap required, opened a new log
I20251027 14:51:26.885480 6159 ts_tablet_manager.cc:1403] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Time spent bootstrapping tablet: real 0.008s user 0.001s sys 0.005s
I20251027 14:51:26.885597 6160 tablet_bootstrap.cc:654] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:26.886216 6161 tablet_bootstrap.cc:654] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:26.886860 6160 log.cc:826] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:26.887388 6161 log.cc:826] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:26.889724 6160 tablet_bootstrap.cc:492] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: No bootstrap required, opened a new log
I20251027 14:51:26.889971 6160 ts_tablet_manager.cc:1403] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Time spent bootstrapping tablet: real 0.008s user 0.002s sys 0.004s
I20251027 14:51:26.890437 6161 tablet_bootstrap.cc:492] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: No bootstrap required, opened a new log
I20251027 14:51:26.890712 6161 ts_tablet_manager.cc:1403] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: Time spent bootstrapping tablet: real 0.010s user 0.000s sys 0.005s
I20251027 14:51:26.891619 6159 raft_consensus.cc:359] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:26.892155 6159 raft_consensus.cc:385] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:26.892313 6159 raft_consensus.cc:740] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6dab693e2e7d4bb1b7a673d3a1a028b5, State: Initialized, Role: FOLLOWER
I20251027 14:51:26.893131 6159 consensus_queue.cc:260] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:26.895498 6160 raft_consensus.cc:359] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:26.895807 6160 raft_consensus.cc:385] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:26.895959 6160 raft_consensus.cc:740] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 573c0f9118c048caab6a512fda2a1b9f, State: Initialized, Role: FOLLOWER
I20251027 14:51:26.896544 6141 heartbeater.cc:499] Master 127.0.104.62:38377 was elected leader, sending a full tablet report...
I20251027 14:51:26.897228 6159 ts_tablet_manager.cc:1434] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
I20251027 14:51:26.897785 6160 consensus_queue.cc:260] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:26.897794 6161 raft_consensus.cc:359] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:26.898169 6161 raft_consensus.cc:385] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:26.898355 6161 raft_consensus.cc:740] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0482873b7ca743a1ae24cf1a6fad151e, State: Initialized, Role: FOLLOWER
I20251027 14:51:26.898762 6011 heartbeater.cc:499] Master 127.0.104.62:38377 was elected leader, sending a full tablet report...
I20251027 14:51:26.899116 6161 consensus_queue.cc:260] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:26.899390 6160 ts_tablet_manager.cc:1434] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Time spent starting tablet: real 0.009s user 0.008s sys 0.001s
I20251027 14:51:26.900319 5881 heartbeater.cc:499] Master 127.0.104.62:38377 was elected leader, sending a full tablet report...
I20251027 14:51:26.902554 6161 ts_tablet_manager.cc:1434] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: Time spent starting tablet: real 0.012s user 0.011s sys 0.000s
W20251027 14:51:26.965761 5882 tablet.cc:2378] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:51:27.052873 6142 tablet.cc:2378] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:51:27.128031 6012 tablet.cc:2378] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:51:27.163220 6166 raft_consensus.cc:493] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:51:27.163556 6166 raft_consensus.cc:515] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:27.165817 6166 leader_election.cc:290] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0482873b7ca743a1ae24cf1a6fad151e (127.0.104.1:39103), 6dab693e2e7d4bb1b7a673d3a1a028b5 (127.0.104.3:34843)
I20251027 14:51:27.169186 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.166103 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:47661 (local address 127.0.104.1:39103)
1027 14:51:27.166278 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:27.166284 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:27.166993 (+ 709us) server_negotiation.cc:408] Connection header received
1027 14:51:27.167161 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:27.167168 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:27.167237 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:27.167354 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:27.168559 (+ 1205us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:27.168565 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:27.168571 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:27.168588 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:27.168624 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:27.168638 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:27.168735 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:27.168895 (+ 160us) server_negotiation.cc:300] Negotiation successful
1027 14:51:27.168945 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:51:27.169847 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.166054 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.104.2:47661)
1027 14:51:27.166885 (+ 831us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.166904 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:27.167039 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:27.167400 (+ 361us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:27.167404 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:27.167445 (+ 41us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:27.168256 (+ 811us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:27.168260 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:27.168279 (+ 19us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:27.168320 (+ 41us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:27.168781 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:27.168786 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:27.168791 (+ 5us) client_negotiation.cc:769] Sending connection context
1027 14:51:27.169449 (+ 658us) client_negotiation.cc:240] Negotiation successful
1027 14:51:27.169506 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":714,"threads_started":1}
I20251027 14:51:27.172150 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.166793 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:46631 (local address 127.0.104.3:34843)
1027 14:51:27.166953 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:27.166959 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:27.170150 (+ 3191us) server_negotiation.cc:408] Connection header received
1027 14:51:27.170338 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:27.170344 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:27.170401 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:27.170460 (+ 59us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:27.171510 (+ 1050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:27.171518 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:27.171523 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:27.171539 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:27.171574 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:27.171588 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:27.171666 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:27.171848 (+ 182us) server_negotiation.cc:300] Negotiation successful
1027 14:51:27.171903 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:51:27.172761 6169 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.166340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34843 (local address 127.0.104.2:46631)
1027 14:51:27.170069 (+ 3729us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.170085 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:27.170221 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:27.170636 (+ 415us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:27.170641 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:27.170662 (+ 21us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:27.171289 (+ 627us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:51:27.171293 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:27.171311 (+ 18us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:51:27.171347 (+ 36us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:27.171738 (+ 391us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:27.171743 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:27.171748 (+ 5us) client_negotiation.cc:769] Sending connection context
1027 14:51:27.172472 (+ 724us) client_negotiation.cc:240] Negotiation successful
1027 14:51:27.172527 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3652,"thread_start_us":609,"threads_started":1}
I20251027 14:51:27.173769 6096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51341b68073b445ba7ab13244f440cee" candidate_uuid: "573c0f9118c048caab6a512fda2a1b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" is_pre_election: true
I20251027 14:51:27.174255 6096 raft_consensus.cc:2468] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573c0f9118c048caab6a512fda2a1b9f in term 0.
I20251027 14:51:27.175077 5900 leader_election.cc:304] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 573c0f9118c048caab6a512fda2a1b9f, 6dab693e2e7d4bb1b7a673d3a1a028b5; no voters:
I20251027 14:51:27.175504 6166 raft_consensus.cc:2804] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:51:27.175642 6166 raft_consensus.cc:493] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:51:27.175734 6166 raft_consensus.cc:3060] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:27.178149 6166 raft_consensus.cc:515] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:27.178711 6166 leader_election.cc:290] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [CANDIDATE]: Term 1 election: Requested vote from peers 0482873b7ca743a1ae24cf1a6fad151e (127.0.104.1:39103), 6dab693e2e7d4bb1b7a673d3a1a028b5 (127.0.104.3:34843)
I20251027 14:51:27.178995 5836 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51341b68073b445ba7ab13244f440cee" candidate_uuid: "573c0f9118c048caab6a512fda2a1b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0482873b7ca743a1ae24cf1a6fad151e" is_pre_election: true
I20251027 14:51:27.179180 5835 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51341b68073b445ba7ab13244f440cee" candidate_uuid: "573c0f9118c048caab6a512fda2a1b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0482873b7ca743a1ae24cf1a6fad151e"
I20251027 14:51:27.179279 6096 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "51341b68073b445ba7ab13244f440cee" candidate_uuid: "573c0f9118c048caab6a512fda2a1b9f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5"
I20251027 14:51:27.179455 6096 raft_consensus.cc:3060] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:27.179590 5836 raft_consensus.cc:2468] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 573c0f9118c048caab6a512fda2a1b9f in term 0.
I20251027 14:51:27.182070 6096 raft_consensus.cc:2468] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 573c0f9118c048caab6a512fda2a1b9f in term 1.
I20251027 14:51:27.182410 5900 leader_election.cc:304] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 573c0f9118c048caab6a512fda2a1b9f, 6dab693e2e7d4bb1b7a673d3a1a028b5; no voters: 0482873b7ca743a1ae24cf1a6fad151e
I20251027 14:51:27.182685 6166 raft_consensus.cc:2804] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:27.183087 6166 raft_consensus.cc:697] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 1 LEADER]: Becoming Leader. State: Replica: 573c0f9118c048caab6a512fda2a1b9f, State: Running, Role: LEADER
I20251027 14:51:27.183609 6166 consensus_queue.cc:237] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } }
I20251027 14:51:27.188316 5695 catalog_manager.cc:5649] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f reported cstate change: term changed from 0 to 1, leader changed from <none> to 573c0f9118c048caab6a512fda2a1b9f (127.0.104.2). New cstate: current_term: 1 leader_uuid: "573c0f9118c048caab6a512fda2a1b9f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "573c0f9118c048caab6a512fda2a1b9f" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 41445 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 } health_report { overall_health: UNKNOWN } } }
I20251027 14:51:27.227276 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.225360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50386 (local address 127.0.104.2:41445)
1027 14:51:27.225535 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:27.225540 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:27.225558 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:51:27.225737 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:27.225741 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:27.225818 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:27.225901 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:27.226719 (+ 818us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:27.226724 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:27.226728 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:27.226743 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:27.226776 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:27.226788 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:27.226886 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:27.226962 (+ 76us) server_negotiation.cc:300] Negotiation successful
1027 14:51:27.227006 (+ 44us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:51:27.235538 5835 raft_consensus.cc:3060] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:27.235736 6096 raft_consensus.cc:1275] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 1 FOLLOWER]: Refusing update from remote peer 573c0f9118c048caab6a512fda2a1b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:51:27.236652 6166 consensus_queue.cc:1048] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "6dab693e2e7d4bb1b7a673d3a1a028b5" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:51:27.238557 5835 raft_consensus.cc:1275] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e [term 1 FOLLOWER]: Refusing update from remote peer 573c0f9118c048caab6a512fda2a1b9f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:51:27.239398 6166 consensus_queue.cc:1048] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [LEADER]: Connected to new peer: Peer: permanent_uuid: "0482873b7ca743a1ae24cf1a6fad151e" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39103 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:51:27.246536 6173 mvcc.cc:204] Tried to move back new op lower bound from 7215418110906724352 to 7215418110708494336. Current Snapshot: MvccSnapshot[applied={T|T < 7215418110906724352}]
I20251027 14:51:27.261981 5695 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:57670:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:51:27.262225 5695 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:57670:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:51:27.266230 5695 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b570e6d6f0884810a31fc7a5c37be2aa: Sending DeleteTablet for 3 replicas of tablet 51341b68073b445ba7ab13244f440cee
I20251027 14:51:27.267627 6076 tablet_service.cc:1552] Processing DeleteTablet for tablet 51341b68073b445ba7ab13244f440cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:27 UTC) from {username='slave'} at 127.0.0.1:56616
I20251027 14:51:27.267835 5815 tablet_service.cc:1552] Processing DeleteTablet for tablet 51341b68073b445ba7ab13244f440cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:27 UTC) from {username='slave'} at 127.0.0.1:35646
I20251027 14:51:27.268167 5816 tablet_service.cc:1552] Processing DeleteTablet for tablet 51341b68073b445ba7ab13244f440cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:27 UTC) from {username='slave'} at 127.0.0.1:35646
I20251027 14:51:27.268919 5946 tablet_service.cc:1552] Processing DeleteTablet for tablet 51341b68073b445ba7ab13244f440cee with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:51:27 UTC) from {username='slave'} at 127.0.0.1:50380
I20251027 14:51:27.269829 6185 tablet_replica.cc:333] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: stopping tablet replica
I20251027 14:51:27.270135 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5754
I20251027 14:51:27.270339 6185 raft_consensus.cc:2243] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:51:27.270962 6185 raft_consensus.cc:2272] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:27.274567 6158 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.272136 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35662 (local address 127.0.104.1:39103)
1027 14:51:27.272322 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:27.272328 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:27.272347 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:51:27.272438 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:27.272444 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:27.272541 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:27.272639 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:27.273784 (+ 1145us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:27.273791 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:27.273797 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:27.273816 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:27.273858 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:27.273875 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:27.273958 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:27.274163 (+ 205us) server_negotiation.cc:300] Negotiation successful
1027 14:51:27.274220 (+ 57us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:51:27.275476 5876 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:27.278596 6185 ts_tablet_manager.cc:1916] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:27.295621 6183 tablet_replica.cc:333] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: stopping tablet replica
I20251027 14:51:27.296113 6183 raft_consensus.cc:2243] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:51:27.296662 6183 raft_consensus.cc:2272] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:27.298547 6185 ts_tablet_manager.cc:1929] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:51:27.298727 6185 log.cc:1199] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/51341b68073b445ba7ab13244f440cee
I20251027 14:51:27.299242 6185 ts_tablet_manager.cc:1950] T 51341b68073b445ba7ab13244f440cee P 573c0f9118c048caab6a512fda2a1b9f: Deleting consensus metadata
I20251027 14:51:27.301254 5679 catalog_manager.cc:4985] TS 573c0f9118c048caab6a512fda2a1b9f (127.0.104.2:41445): tablet 51341b68073b445ba7ab13244f440cee (table test-table [id=aef93b0f0aa44e17aab163ade03125fc]) successfully deleted
I20251027 14:51:27.302738 6183 ts_tablet_manager.cc:1916] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:27.310315 6183 ts_tablet_manager.cc:1929] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:51:27.310544 6183 log.cc:1199] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestDisableAuthenticationEncryption.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/51341b68073b445ba7ab13244f440cee
I20251027 14:51:27.311100 6183 ts_tablet_manager.cc:1950] T 51341b68073b445ba7ab13244f440cee P 6dab693e2e7d4bb1b7a673d3a1a028b5: Deleting consensus metadata
I20251027 14:51:27.313131 5680 catalog_manager.cc:4985] TS 6dab693e2e7d4bb1b7a673d3a1a028b5 (127.0.104.3:34843): tablet 51341b68073b445ba7ab13244f440cee (table test-table [id=aef93b0f0aa44e17aab163ade03125fc]) successfully deleted
I20251027 14:51:27.393617 6184 tablet_replica.cc:333] T 51341b68073b445ba7ab13244f440cee P 0482873b7ca743a1ae24cf1a6fad151e: stopping tablet replica
I20251027 14:51:27.393661 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5754
I20251027 14:51:27.413846 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5884
W20251027 14:51:27.413818 5679 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:39103 (error 108)
I20251027 14:51:27.416177 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.415482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35678)
1027 14:51:27.415836 (+ 354us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.415976 (+ 140us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":260,"thread_start_us":530,"threads_started":1}
W20251027 14:51:27.416594 5679 catalog_manager.cc:4712] TS 0482873b7ca743a1ae24cf1a6fad151e (127.0.104.1:39103): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 51341b68073b445ba7ab13244f440cee: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
I20251027 14:51:27.417011 6154 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.414617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50394 (local address 127.0.104.2:41445)
1027 14:51:27.414802 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:27.414812 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:27.414868 (+ 56us) server_negotiation.cc:408] Connection header received
1027 14:51:27.415034 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:27.415040 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:27.415103 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:27.415204 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:27.416309 (+ 1105us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:27.416314 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:27.416318 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:27.416334 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:27.416369 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:27.416412 (+ 43us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:27.416534 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:27.416680 (+ 146us) server_negotiation.cc:300] Negotiation successful
1027 14:51:27.416734 (+ 54us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:51:27.417681 6006 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:27.461195 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.460657 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35694)
1027 14:51:27.460900 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.461002 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20251027 14:51:27.462342 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.461958 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35698)
1027 14:51:27.462086 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.462168 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20251027 14:51:27.470420 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.470099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35700)
1027 14:51:27.470238 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.470306 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251027 14:51:27.471268 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.470957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35714)
1027 14:51:27.471084 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.471145 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20251027 14:51:27.511495 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.511062 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35726)
1027 14:51:27.511267 (+ 205us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.511350 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":80}
I20251027 14:51:27.512532 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.512099 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35728)
1027 14:51:27.512277 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.512340 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20251027 14:51:27.523126 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5884
I20251027 14:51:27.537164 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6014
I20251027 14:51:27.539772 6157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.537909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56630 (local address 127.0.104.3:34843)
1027 14:51:27.538068 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:27.538073 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:27.538090 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:51:27.538246 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:27.538273 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:27.538342 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:27.538422 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:27.539196 (+ 774us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:27.539202 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:27.539205 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:27.539217 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:27.539287 (+ 70us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:27.539299 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:27.539386 (+ 87us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:27.539461 (+ 75us) server_negotiation.cc:300] Negotiation successful
1027 14:51:27.539503 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251027 14:51:27.540369 6136 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:27.548182 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.547644 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35734)
1027 14:51:27.547905 (+ 261us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.548004 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":67}
I20251027 14:51:27.549258 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.548938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35750)
1027 14:51:27.549063 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.549125 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251027 14:51:27.613582 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.613142 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35752)
1027 14:51:27.613348 (+ 206us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.613433 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20251027 14:51:27.614599 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.614209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35766)
1027 14:51:27.614365 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.614448 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20251027 14:51:27.626240 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6014
I20251027 14:51:27.639375 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 5664
I20251027 14:51:27.640518 5725 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:27.741009 6189 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:27.740594 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39103 (local address 127.0.0.1:35782)
1027 14:51:27.740757 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:27.740843 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:39103: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251027 14:51:27.741305 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 5664
2025-10-27T14:51:27Z chronyd exiting
[ OK ] SecurityITest.TestDisableAuthenticationEncryption (2734 ms)
[ RUN ] SecurityITest.TestJwtMiniCluster
I20251027 14:51:27.778028 416 mini_oidc.cc:150] Starting JWKS server
I20251027 14:51:27.780599 416 webserver.cc:492] Webserver started at https://127.0.0.1:42031/ using document root <none> and password file <none>
I20251027 14:51:27.780882 416 mini_oidc.cc:186] Starting OIDC Discovery server
I20251027 14:51:27.781407 416 webserver.cc:492] Webserver started at http://127.0.0.1:39085/ using document root <none> and password file <none>
2025-10-27T14:51:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:27Z Disabled control of system clock
I20251027 14:51:27.806016 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39345
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:34517
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:39345
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42031/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/testchainca.cert with env {}
W20251027 14:51:28.115109 6198 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:28.115455 6198 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:28.115583 6198 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:28.123492 6198 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:28.123594 6198 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:28.123659 6198 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:28.123719 6198 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:28.123800 6198 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251027 14:51:28.123860 6198 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251027 14:51:28.123929 6198 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:42031/jwks.json
I20251027 14:51:28.134315 6198 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34517
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:39345
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39345
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:42031/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:28.135910 6198 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:28.137239 6198 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:28.145249 6204 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:28.145366 6206 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:28.145279 6203 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:28.146083 6198 server_base.cc:1047] running on GCE node
I20251027 14:51:28.146795 6198 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:28.148128 6198 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:28.149340 6198 hybrid_clock.cc:648] HybridClock initialized: now 1761576688149276 us; error 36 us; skew 500 ppm
I20251027 14:51:28.149643 6198 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:28.151623 6198 webserver.cc:492] Webserver started at http://127.0.104.62:42301/ using document root <none> and password file <none>
I20251027 14:51:28.152284 6198 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:28.152413 6198 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:28.152724 6198 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:28.155107 6198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "3905edbba42544aaa42c2a8827754d67"
format_stamp: "Formatted at 2025-10-27 14:51:28 on dist-test-slave-hxxt"
server_key: "62003529d45af804c5093cef5388276c"
server_key_iv: "586ec9e989aa4c0023a2be7009ba7af7"
server_key_version: "encryptionkey@0"
I20251027 14:51:28.155886 6198 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "3905edbba42544aaa42c2a8827754d67"
format_stamp: "Formatted at 2025-10-27 14:51:28 on dist-test-slave-hxxt"
server_key: "62003529d45af804c5093cef5388276c"
server_key_iv: "586ec9e989aa4c0023a2be7009ba7af7"
server_key_version: "encryptionkey@0"
I20251027 14:51:28.161051 6198 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20251027 14:51:28.164489 6212 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:28.165831 6198 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:28.165987 6198 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "3905edbba42544aaa42c2a8827754d67"
format_stamp: "Formatted at 2025-10-27 14:51:28 on dist-test-slave-hxxt"
server_key: "62003529d45af804c5093cef5388276c"
server_key_iv: "586ec9e989aa4c0023a2be7009ba7af7"
server_key_version: "encryptionkey@0"
I20251027 14:51:28.166199 6198 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:28.183712 6198 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:28.184598 6198 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:28.184863 6198 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:28.202824 6198 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39345
I20251027 14:51:28.202842 6263 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39345 every 8 connection(s)
I20251027 14:51:28.204280 6198 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:28.208527 6264 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:28.211439 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6198
I20251027 14:51:28.211611 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:28.212148 416 external_mini_cluster.cc:1468] Setting key 482a1f03fe70d22eef2316c579a20d46
I20251027 14:51:28.221246 6264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67: Bootstrap starting.
I20251027 14:51:28.225201 6264 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:28.225279 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:28.214070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47850 (local address 127.0.104.62:39345)
1027 14:51:28.215132 (+ 1062us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:28.215154 (+ 22us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:28.215205 (+ 51us) server_negotiation.cc:408] Connection header received
1027 14:51:28.216724 (+ 1519us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:28.216753 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:28.217430 (+ 677us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:28.217956 (+ 526us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:28.218623 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.220559 (+ 1936us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:28.221676 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.222180 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:28.222495 (+ 315us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:28.222597 (+ 102us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:28.222617 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:28.223350 (+ 733us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:28.223417 (+ 67us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:28.223442 (+ 25us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:28.223552 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:28.223752 (+ 200us) server_negotiation.cc:300] Negotiation successful
1027 14:51:28.224054 (+ 302us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":379,"threads_started":1}
I20251027 14:51:28.226342 6264 log.cc:826] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:28.229301 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:51:28.229779 6264 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67: No bootstrap required, opened a new log
I20251027 14:51:28.235594 6264 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3905edbba42544aaa42c2a8827754d67" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39345 } }
I20251027 14:51:28.235996 6264 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:28.236119 6264 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3905edbba42544aaa42c2a8827754d67, State: Initialized, Role: FOLLOWER
I20251027 14:51:28.236712 6264 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3905edbba42544aaa42c2a8827754d67" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39345 } }
I20251027 14:51:28.236902 6264 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:28.237044 6264 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:28.237205 6264 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:28.239476 6264 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3905edbba42544aaa42c2a8827754d67" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39345 } }
I20251027 14:51:28.239918 6264 leader_election.cc:304] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3905edbba42544aaa42c2a8827754d67; no voters:
I20251027 14:51:28.240502 6264 leader_election.cc:290] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:28.240608 6269 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:28.241520 6269 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [term 1 LEADER]: Becoming Leader. State: Replica: 3905edbba42544aaa42c2a8827754d67, State: Running, Role: LEADER
I20251027 14:51:28.242024 6269 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3905edbba42544aaa42c2a8827754d67" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39345 } }
I20251027 14:51:28.242501 6264 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:28.244830 6271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3905edbba42544aaa42c2a8827754d67. Latest consensus state: current_term: 1 leader_uuid: "3905edbba42544aaa42c2a8827754d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3905edbba42544aaa42c2a8827754d67" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39345 } } }
I20251027 14:51:28.245172 6271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:28.245604 6270 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3905edbba42544aaa42c2a8827754d67" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3905edbba42544aaa42c2a8827754d67" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39345 } } }
I20251027 14:51:28.245849 6270 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:28.249634 6276 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:28.255551 6276 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:28.265436 6276 catalog_manager.cc:1357] Generated new cluster ID: ffcd66b87d584e5c852bf34217986f33
I20251027 14:51:28.265563 6276 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:28.281308 6276 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:28.282958 6276 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:28.304169 6276 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3905edbba42544aaa42c2a8827754d67: Generated new TSK 0
I20251027 14:51:28.305361 6276 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:51:28.358461 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:28.328870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47858 (local address 127.0.104.62:39345)
1027 14:51:28.329064 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:28.329072 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:28.329525 (+ 453us) server_negotiation.cc:408] Connection header received
1027 14:51:28.329825 (+ 300us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:28.329830 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:28.329900 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:28.330017 (+ 117us) server_negotiation.cc:227] Negotiated authn=JWT
1027 14:51:28.330873 (+ 856us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.331796 (+ 923us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:28.333490 (+ 1694us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.333905 (+ 415us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:28.334052 (+ 147us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1027 14:51:28.357719 (+ 23667us) server_negotiation.cc:378] Sending JWT_EXCHANGE NegotiatePB response
1027 14:51:28.357820 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:28.358069 (+ 249us) server_negotiation.cc:300] Negotiation successful
1027 14:51:28.358162 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62,"thread_start_us":318,"threads_started":1}
I20251027 14:51:28.377488 6267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:28.370335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47874 (local address 127.0.104.62:39345)
1027 14:51:28.370521 (+ 186us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:28.370528 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:28.370945 (+ 417us) server_negotiation.cc:408] Connection header received
1027 14:51:28.371124 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:28.371131 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:28.371227 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:28.371342 (+ 115us) server_negotiation.cc:227] Negotiated authn=JWT
1027 14:51:28.372031 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.373464 (+ 1433us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:28.374551 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.375074 (+ 523us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:28.375152 (+ 78us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1027 14:51:28.376857 (+ 1705us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1027 14:51:28.377185 (+ 328us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:47874: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"server-negotiator.queue_time_us":45}
W20251027 14:51:28.377601 6267 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:47874: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
W20251027 14:51:28.378649 6300 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:28.370162 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39345 (local address 127.0.0.1:47874)
1027 14:51:28.370850 (+ 688us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:28.370865 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:28.370991 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:28.371357 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:28.371360 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:28.371370 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
1027 14:51:28.371895 (+ 525us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:28.371907 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.373583 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:28.373587 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:28.374418 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:28.374430 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:28.374547 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:28.374576 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1027 14:51:28.377086 (+ 2510us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
1027 14:51:28.377302 (+ 216us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.104.62:39345: FATAL_INVALID_JWT: Not authorized: JWT verification failed: failed to verify signature: VerifyFinal failed (error:0407008A:rsa routines:RSA_padding_check_PKCS1_type_1:invalid padding:../crypto/rsa/rsa_pk1.c:67 error:04067072:rsa routines:rsa_ossl_public_decrypt:padding check failed:../crypto/rsa/rsa_ossl.c:581)
Metrics: {"client-negotiator.queue_time_us":558,"thread_start_us":461,"threads_started":1}
I20251027 14:51:43.394937 6306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:43.389910 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58026 (local address 127.0.104.62:39345)
1027 14:51:43.390355 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:43.390360 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:43.390448 (+ 88us) server_negotiation.cc:408] Connection header received
1027 14:51:43.390655 (+ 207us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:43.390660 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:43.390726 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:43.390830 (+ 104us) server_negotiation.cc:227] Negotiated authn=JWT
1027 14:51:43.391412 (+ 582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:43.392325 (+ 913us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:43.393412 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:43.393712 (+ 300us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:43.393765 (+ 53us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1027 14:51:43.394570 (+ 805us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1027 14:51:43.394751 (+ 181us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:58026: JWT verification failed: token expired
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":266,"threads_started":1}
W20251027 14:51:43.395022 6306 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:58026: JWT verification failed: token expired
W20251027 14:51:43.395076 6305 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:43.389795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39345 (local address 127.0.0.1:58026)
1027 14:51:43.390357 (+ 562us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:43.390374 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:43.390506 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:43.390855 (+ 349us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:43.390858 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:43.390868 (+ 10us) client_negotiation.cc:189] Negotiated authn=JWT
1027 14:51:43.391281 (+ 413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:43.391291 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:43.392493 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:43.392496 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:43.393303 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:43.393314 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:43.393436 (+ 122us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:43.393465 (+ 29us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1027 14:51:43.394700 (+ 1235us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
1027 14:51:43.394862 (+ 162us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.104.62:39345: FATAL_INVALID_JWT: Not authorized: JWT verification failed: token expired
Metrics: {"client-negotiator.queue_time_us":423,"thread_start_us":280,"threads_started":1}
W20251027 14:51:43.400074 6306 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:43.398854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58032 (local address 127.0.104.62:39345)
1027 14:51:43.398993 (+ 139us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:43.398999 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:43.399270 (+ 271us) server_negotiation.cc:408] Connection header received
1027 14:51:43.399426 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:43.399430 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:43.399489 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:43.399562 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:43.399900 (+ 338us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58032: BlockingRecv error: recv got EOF from 127.0.0.1:58032 (error 108)
Metrics: {"server-negotiator.queue_time_us":38}
W20251027 14:51:43.405339 6316 client_negotiation.cc:375] the client has a JWT but it isn't advertising its JWT-based authn capability since it doesn't trust the certificate of the server at 127.0.104.62:39345
W20251027 14:51:43.406114 6306 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:43.404923 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58040 (local address 127.0.104.62:39345)
1027 14:51:43.405114 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:43.405119 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:43.405257 (+ 138us) server_negotiation.cc:408] Connection header received
1027 14:51:43.405532 (+ 275us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:43.405536 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:43.405606 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:43.405685 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:43.405972 (+ 287us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58040: BlockingRecv error: recv got EOF from 127.0.0.1:58040 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:51:43.406615 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6198
I20251027 14:51:43.407686 6259 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:43.509256 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6198
2025-10-27T14:51:43Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniCluster (15786 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithInvalidCert
I20251027 14:51:43.563712 416 mini_oidc.cc:150] Starting JWKS server
I20251027 14:51:43.565606 416 webserver.cc:492] Webserver started at https://127.0.0.1:34357/ using document root <none> and password file <none>
I20251027 14:51:43.565798 416 mini_oidc.cc:186] Starting OIDC Discovery server
I20251027 14:51:43.566282 416 webserver.cc:492] Webserver started at http://127.0.0.1:33061/ using document root <none> and password file <none>
2025-10-27T14:51:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:43Z Disabled control of system clock
I20251027 14:51:43.591921 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43757
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:45879
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43757
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34357/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/testchainca.cert with env {}
W20251027 14:51:43.911729 6323 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:43.912072 6323 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:43.912196 6323 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:43.920281 6323 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:43.920413 6323 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:43.920488 6323 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:43.920550 6323 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:43.920630 6323 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251027 14:51:43.920691 6323 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251027 14:51:43.920758 6323 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:34357/jwks.json
I20251027 14:51:43.931236 6323 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45879
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43757
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43757
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:34357/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:43.932796 6323 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:43.934096 6323 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:43.941982 6329 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:43.942220 6331 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:43.942456 6328 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:43.942782 6323 server_base.cc:1047] running on GCE node
I20251027 14:51:43.943506 6323 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:43.944881 6323 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:43.946087 6323 hybrid_clock.cc:648] HybridClock initialized: now 1761576703946033 us; error 35 us; skew 500 ppm
I20251027 14:51:43.946429 6323 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:43.948469 6323 webserver.cc:492] Webserver started at http://127.0.104.62:42255/ using document root <none> and password file <none>
I20251027 14:51:43.949151 6323 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:43.949270 6323 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:43.949568 6323 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:43.952023 6323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "69e180813c96451a9842513697c29808"
format_stamp: "Formatted at 2025-10-27 14:51:43 on dist-test-slave-hxxt"
server_key: "cc6d6c7d789c97735ff800741142bfb1"
server_key_iv: "61408c17b919c197c34d5c0e4bd3ec05"
server_key_version: "encryptionkey@0"
I20251027 14:51:43.952768 6323 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "69e180813c96451a9842513697c29808"
format_stamp: "Formatted at 2025-10-27 14:51:43 on dist-test-slave-hxxt"
server_key: "cc6d6c7d789c97735ff800741142bfb1"
server_key_iv: "61408c17b919c197c34d5c0e4bd3ec05"
server_key_version: "encryptionkey@0"
I20251027 14:51:43.957602 6323 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:51:43.960757 6337 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:43.962025 6323 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:51:43.962177 6323 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "69e180813c96451a9842513697c29808"
format_stamp: "Formatted at 2025-10-27 14:51:43 on dist-test-slave-hxxt"
server_key: "cc6d6c7d789c97735ff800741142bfb1"
server_key_iv: "61408c17b919c197c34d5c0e4bd3ec05"
server_key_version: "encryptionkey@0"
I20251027 14:51:43.962355 6323 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:43.986143 6323 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:43.987004 6323 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:43.987287 6323 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:44.005018 6323 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43757
I20251027 14:51:44.005048 6388 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43757 every 8 connection(s)
I20251027 14:51:44.006429 6323 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:44.010075 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6323
I20251027 14:51:44.010268 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithInvalidCert.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:44.010490 6389 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:44.010809 416 external_mini_cluster.cc:1468] Setting key e647465752b6bd5975d22a5e3b68959b
I20251027 14:51:44.022256 6389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808: Bootstrap starting.
I20251027 14:51:44.025446 6389 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:44.026573 6389 log.cc:826] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:44.027073 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:44.012540 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53428 (local address 127.0.104.62:43757)
1027 14:51:44.013431 (+ 891us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:44.013440 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:44.013478 (+ 38us) server_negotiation.cc:408] Connection header received
1027 14:51:44.015080 (+ 1602us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:44.015109 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:44.015575 (+ 466us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:44.015948 (+ 373us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:44.016788 (+ 840us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.018131 (+ 1343us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:44.019671 (+ 1540us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.020130 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:44.024644 (+ 4514us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:44.024731 (+ 87us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:44.024747 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:44.025287 (+ 540us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:44.025334 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:44.025357 (+ 23us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:44.025459 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:44.025784 (+ 325us) server_negotiation.cc:300] Negotiation successful
1027 14:51:44.026103 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":497,"thread_start_us":330,"threads_started":1}
I20251027 14:51:44.029680 6389 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808: No bootstrap required, opened a new log
I20251027 14:51:44.030915 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:51:44.034950 6389 raft_consensus.cc:359] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e180813c96451a9842513697c29808" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43757 } }
I20251027 14:51:44.035269 6389 raft_consensus.cc:385] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:44.035365 6389 raft_consensus.cc:740] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e180813c96451a9842513697c29808, State: Initialized, Role: FOLLOWER
I20251027 14:51:44.035889 6389 consensus_queue.cc:260] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e180813c96451a9842513697c29808" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43757 } }
I20251027 14:51:44.036082 6389 raft_consensus.cc:399] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:44.036192 6389 raft_consensus.cc:493] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:44.036334 6389 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:44.038648 6389 raft_consensus.cc:515] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e180813c96451a9842513697c29808" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43757 } }
I20251027 14:51:44.039109 6389 leader_election.cc:304] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 69e180813c96451a9842513697c29808; no voters:
I20251027 14:51:44.039610 6389 leader_election.cc:290] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:44.039716 6394 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:44.040899 6394 raft_consensus.cc:697] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [term 1 LEADER]: Becoming Leader. State: Replica: 69e180813c96451a9842513697c29808, State: Running, Role: LEADER
I20251027 14:51:44.041452 6394 consensus_queue.cc:237] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e180813c96451a9842513697c29808" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43757 } }
I20251027 14:51:44.041606 6389 sys_catalog.cc:565] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:44.044235 6396 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 69e180813c96451a9842513697c29808. Latest consensus state: current_term: 1 leader_uuid: "69e180813c96451a9842513697c29808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e180813c96451a9842513697c29808" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43757 } } }
I20251027 14:51:44.044641 6395 sys_catalog.cc:455] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "69e180813c96451a9842513697c29808" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "69e180813c96451a9842513697c29808" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43757 } } }
I20251027 14:51:44.044945 6396 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:44.045266 6395 sys_catalog.cc:458] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:44.047822 6404 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:44.052850 6404 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:44.061309 6404 catalog_manager.cc:1357] Generated new cluster ID: 5c72c87095a44c1aaf66e9d9482f3046
I20251027 14:51:44.061416 6404 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:44.094995 6404 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:44.096195 6404 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:44.106868 6404 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 69e180813c96451a9842513697c29808: Generated new TSK 0
I20251027 14:51:44.107698 6404 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251027 14:51:44.140746 6419 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251027 14:51:44.140857 6419 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251027 14:51:44.142129 6392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:44.121501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53440 (local address 127.0.104.62:43757)
1027 14:51:44.121660 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:44.121667 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:44.122170 (+ 503us) server_negotiation.cc:408] Connection header received
1027 14:51:44.122362 (+ 192us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:44.122369 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:44.122468 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:44.122639 (+ 171us) server_negotiation.cc:227] Negotiated authn=JWT
1027 14:51:44.123282 (+ 643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.124680 (+ 1398us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:44.125937 (+ 1257us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.126481 (+ 544us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:44.126666 (+ 185us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1027 14:51:44.141421 (+ 14755us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1027 14:51:44.141798 (+ 377us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:53440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":47}
W20251027 14:51:44.142191 6418 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:44.121468 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43757 (local address 127.0.0.1:53440)
1027 14:51:44.122067 (+ 599us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:44.122083 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:44.122211 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:44.122622 (+ 411us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:44.122626 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:44.122635 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1027 14:51:44.123143 (+ 508us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:44.123154 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.124887 (+ 1733us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:44.124892 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:44.125790 (+ 898us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:44.125806 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.125952 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:44.125983 (+ 31us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1027 14:51:44.141687 (+ 15704us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1027 14:51:44.141895 (+ 208us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.104.62:43757: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":452,"thread_start_us":320,"threads_started":1}
W20251027 14:51:44.142256 6392 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:53440: Failed to load JWKS: Error downloading JWKS from 'https://localhost:34357/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20251027 14:51:44.143515 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6323
I20251027 14:51:44.144613 6384 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:44.235356 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6323
2025-10-27T14:51:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithInvalidCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithUntrustedCert
I20251027 14:51:44.287767 416 mini_oidc.cc:150] Starting JWKS server
I20251027 14:51:44.289708 416 webserver.cc:492] Webserver started at https://127.0.0.1:43463/ using document root <none> and password file <none>
I20251027 14:51:44.289944 416 mini_oidc.cc:186] Starting OIDC Discovery server
I20251027 14:51:44.290474 416 webserver.cc:492] Webserver started at http://127.0.0.1:35231/ using document root <none> and password file <none>
2025-10-27T14:51:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:44Z Disabled control of system clock
I20251027 14:51:44.315953 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42455
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35983
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42455
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43463/jwks.json
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251027 14:51:44.628546 6426 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:44.628868 6426 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:44.628960 6426 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:44.636999 6426 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:44.637096 6426 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:44.637142 6426 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:44.637188 6426 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:44.637243 6426 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251027 14:51:44.637287 6426 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251027 14:51:44.637334 6426 flags.cc:425] Enabled experimental flag: --jwks_url=https://localhost:43463/jwks.json
I20251027 14:51:44.647023 6426 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35983
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42455
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42455
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=https://localhost:43463/jwks.json
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:44.648680 6426 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:44.650415 6426 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:44.657883 6431 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:44.658083 6434 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:44.658172 6432 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:44.659025 6426 server_base.cc:1047] running on GCE node
I20251027 14:51:44.659720 6426 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:44.661063 6426 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:44.662279 6426 hybrid_clock.cc:648] HybridClock initialized: now 1761576704662219 us; error 44 us; skew 500 ppm
I20251027 14:51:44.662590 6426 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:44.664615 6426 webserver.cc:492] Webserver started at http://127.0.104.62:36601/ using document root <none> and password file <none>
I20251027 14:51:44.665316 6426 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:44.665433 6426 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:44.665743 6426 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:44.668329 6426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c"
format_stamp: "Formatted at 2025-10-27 14:51:44 on dist-test-slave-hxxt"
server_key: "56ef5b1321679f2186b897e71188db2c"
server_key_iv: "934fd663c9ca5ef57f23a0ab62c1343c"
server_key_version: "encryptionkey@0"
I20251027 14:51:44.669108 6426 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c"
format_stamp: "Formatted at 2025-10-27 14:51:44 on dist-test-slave-hxxt"
server_key: "56ef5b1321679f2186b897e71188db2c"
server_key_iv: "934fd663c9ca5ef57f23a0ab62c1343c"
server_key_version: "encryptionkey@0"
I20251027 14:51:44.674204 6426 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:51:44.677549 6440 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:44.678812 6426 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:51:44.678958 6426 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c"
format_stamp: "Formatted at 2025-10-27 14:51:44 on dist-test-slave-hxxt"
server_key: "56ef5b1321679f2186b897e71188db2c"
server_key_iv: "934fd663c9ca5ef57f23a0ab62c1343c"
server_key_version: "encryptionkey@0"
I20251027 14:51:44.679178 6426 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:44.692025 6426 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:44.692867 6426 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:44.693120 6426 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:44.710988 6426 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42455
I20251027 14:51:44.711010 6491 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42455 every 8 connection(s)
I20251027 14:51:44.712428 6426 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:44.713402 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6426
I20251027 14:51:44.713599 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithUntrustedCert.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:44.714314 416 external_mini_cluster.cc:1468] Setting key 7cc571390b4db50bac92bdcd3ba2f106
I20251027 14:51:44.717757 6492 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:44.725965 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:44.716268 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50264 (local address 127.0.104.62:42455)
1027 14:51:44.717208 (+ 940us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:44.717224 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:44.717271 (+ 47us) server_negotiation.cc:408] Connection header received
1027 14:51:44.718604 (+ 1333us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:44.718626 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:44.719078 (+ 452us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:44.719489 (+ 411us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:44.720255 (+ 766us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.721643 (+ 1388us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:44.722759 (+ 1116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.723266 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:44.723700 (+ 434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:44.723789 (+ 89us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:44.723803 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:44.724303 (+ 500us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:44.724349 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:44.724367 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:44.724500 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:44.724711 (+ 211us) server_negotiation.cc:300] Negotiation successful
1027 14:51:44.725004 (+ 293us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":539,"thread_start_us":318,"threads_started":1}
I20251027 14:51:44.728133 6492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c: Bootstrap starting.
I20251027 14:51:44.729525 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:51:44.731338 6492 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:44.732487 6492 log.cc:826] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:44.735175 6492 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c: No bootstrap required, opened a new log
I20251027 14:51:44.740198 6492 raft_consensus.cc:359] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42455 } }
I20251027 14:51:44.740567 6492 raft_consensus.cc:385] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:44.740644 6492 raft_consensus.cc:740] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b6f68fae463c4f319e98ff6cd2bc4e1c, State: Initialized, Role: FOLLOWER
I20251027 14:51:44.741259 6492 consensus_queue.cc:260] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42455 } }
I20251027 14:51:44.741454 6492 raft_consensus.cc:399] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:44.741591 6492 raft_consensus.cc:493] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:44.741750 6492 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:44.743861 6492 raft_consensus.cc:515] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42455 } }
I20251027 14:51:44.744311 6492 leader_election.cc:304] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b6f68fae463c4f319e98ff6cd2bc4e1c; no voters:
I20251027 14:51:44.744880 6492 leader_election.cc:290] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:44.745007 6497 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:44.745573 6497 raft_consensus.cc:697] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [term 1 LEADER]: Becoming Leader. State: Replica: b6f68fae463c4f319e98ff6cd2bc4e1c, State: Running, Role: LEADER
I20251027 14:51:44.746054 6497 consensus_queue.cc:237] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42455 } }
I20251027 14:51:44.746888 6492 sys_catalog.cc:565] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:44.748853 6499 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b6f68fae463c4f319e98ff6cd2bc4e1c. Latest consensus state: current_term: 1 leader_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42455 } } }
I20251027 14:51:44.749329 6499 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:44.749269 6498 sys_catalog.cc:455] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b6f68fae463c4f319e98ff6cd2bc4e1c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42455 } } }
I20251027 14:51:44.749503 6498 sys_catalog.cc:458] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:44.753515 6505 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:44.759567 6505 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:44.767868 6505 catalog_manager.cc:1357] Generated new cluster ID: fee388ade3bc44f0a7247aeeca883188
I20251027 14:51:44.767997 6505 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:44.781898 6505 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:44.783588 6505 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:44.794629 6505 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b6f68fae463c4f319e98ff6cd2bc4e1c: Generated new TSK 0
I20251027 14:51:44.795470 6505 catalog_manager.cc:1524] Initializing in-progress tserver states...
E20251027 14:51:44.857635 6522 webserver.cc:578] Webserver: sslize failed: error:14094418:SSL routines:ssl3_read_bytes:tlsv1 alert unknown ca
E20251027 14:51:44.857738 6522 webserver.cc:578] Webserver: error shutting down SSL connection: error:140E0197:SSL routines:SSL_shutdown:shutdown while in init
I20251027 14:51:44.862370 6495 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:44.815038 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50270 (local address 127.0.104.62:42455)
1027 14:51:44.815219 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:44.815226 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:44.815678 (+ 452us) server_negotiation.cc:408] Connection header received
1027 14:51:44.815887 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:44.815894 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:44.815993 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:44.816157 (+ 164us) server_negotiation.cc:227] Negotiated authn=JWT
1027 14:51:44.816808 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.818288 (+ 1480us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:44.819447 (+ 1159us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.819993 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:44.820188 (+ 195us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1027 14:51:44.861678 (+ 41490us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1027 14:51:44.862087 (+ 409us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:50270: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"server-negotiator.queue_time_us":53}
W20251027 14:51:44.862435 6521 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:44.814998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42455 (local address 127.0.0.1:50270)
1027 14:51:44.815577 (+ 579us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:44.815593 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:44.815719 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:44.816155 (+ 436us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:44.816158 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:44.816167 (+ 9us) client_negotiation.cc:189] Negotiated authn=JWT
1027 14:51:44.816664 (+ 497us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:44.816676 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.818421 (+ 1745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:44.818425 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:44.819314 (+ 889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:44.819326 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:44.819471 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:44.819499 (+ 28us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1027 14:51:44.861957 (+ 42458us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
1027 14:51:44.862166 (+ 209us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.104.62:42455: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":318,"threads_started":1}
W20251027 14:51:44.862504 6495 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:50270: Failed to load JWKS: Error downloading JWKS from 'https://localhost:43463/jwks.json': curl error: SSL peer certificate or SSH remote key was not OK: SSL certificate problem: unable to get local issuer certificate
I20251027 14:51:44.863538 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6426
I20251027 14:51:44.864770 6487 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:44.958189 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6426
2025-10-27T14:51:44Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithUntrustedCert (723 ms)
[ RUN ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS
2025-10-27T14:51:45Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:45Z Disabled control of system clock
I20251027 14:51:45.036851 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44299
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:33833
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44299
--enable_jwt_token_auth=true
--jwks_url=default.url
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--trusted_certificate_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/testchainca.cert with env {}
W20251027 14:51:45.355482 6527 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:45.355789 6527 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:45.355880 6527 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:45.363720 6527 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:45.363844 6527 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:45.363914 6527 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:45.363963 6527 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:45.364019 6527 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
W20251027 14:51:45.364064 6527 flags.cc:425] Enabled experimental flag: --enable_jwt_token_auth=true
W20251027 14:51:45.364111 6527 flags.cc:425] Enabled experimental flag: --jwks_url=default.url
I20251027 14:51:45.374368 6527 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33833
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44299
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--enable_jwt_token_auth=true
--jwks_url=default.url
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--trusted_certificate_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/testchainca.cert
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:45.375834 6527 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:45.377076 6527 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:45.384725 6532 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:45.384717 6533 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:45.385403 6535 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:45.385905 6527 server_base.cc:1047] running on GCE node
I20251027 14:51:45.386611 6527 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:45.387989 6527 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:45.389189 6527 hybrid_clock.cc:648] HybridClock initialized: now 1761576705389128 us; error 38 us; skew 500 ppm
I20251027 14:51:45.389497 6527 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:51:45.391651 6527 webserver.cc:492] Webserver started at http://127.0.104.62:42371/ using document root <none> and password file <none>
I20251027 14:51:45.392520 6527 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:45.392622 6527 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:45.392908 6527 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:45.395372 6527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "223723076e8c4c7f95787345a1658b0f"
format_stamp: "Formatted at 2025-10-27 14:51:45 on dist-test-slave-hxxt"
server_key: "07a4a72b83fcb06d3148deaaf9404349"
server_key_iv: "c141638e6f3be61dc496fd3c31820566"
server_key_version: "encryptionkey@0"
I20251027 14:51:45.396085 6527 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "223723076e8c4c7f95787345a1658b0f"
format_stamp: "Formatted at 2025-10-27 14:51:45 on dist-test-slave-hxxt"
server_key: "07a4a72b83fcb06d3148deaaf9404349"
server_key_iv: "c141638e6f3be61dc496fd3c31820566"
server_key_version: "encryptionkey@0"
I20251027 14:51:45.401332 6527 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251027 14:51:45.404536 6541 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:45.405772 6527 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:51:45.405915 6527 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "223723076e8c4c7f95787345a1658b0f"
format_stamp: "Formatted at 2025-10-27 14:51:45 on dist-test-slave-hxxt"
server_key: "07a4a72b83fcb06d3148deaaf9404349"
server_key_iv: "c141638e6f3be61dc496fd3c31820566"
server_key_version: "encryptionkey@0"
I20251027 14:51:45.406107 6527 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:45.428351 6527 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:45.429221 6527 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:45.429471 6527 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:45.447417 6527 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44299
I20251027 14:51:45.447438 6592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44299 every 8 connection(s)
I20251027 14:51:45.448825 6527 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:45.452955 6593 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:45.455540 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6527
I20251027 14:51:45.455722 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:45.456244 416 external_mini_cluster.cc:1468] Setting key 2d8e8d01a9d69a471b62f480d36a6963
I20251027 14:51:45.465312 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f: Bootstrap starting.
I20251027 14:51:45.469372 6593 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:45.470549 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:45.458069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37156 (local address 127.0.104.62:44299)
1027 14:51:45.459061 (+ 992us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:45.459074 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:45.459122 (+ 48us) server_negotiation.cc:408] Connection header received
1027 14:51:45.460631 (+ 1509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:45.460660 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:45.461307 (+ 647us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:45.461867 (+ 560us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:45.462768 (+ 901us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:45.464613 (+ 1845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:45.466382 (+ 1769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:45.466887 (+ 505us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:45.467435 (+ 548us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:45.467544 (+ 109us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:45.467568 (+ 24us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:51:45.468197 (+ 629us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:45.468268 (+ 71us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:51:45.468289 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:45.468786 (+ 497us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:45.468971 (+ 185us) server_negotiation.cc:300] Negotiation successful
1027 14:51:45.469304 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":511,"thread_start_us":336,"threads_started":1}
I20251027 14:51:45.470619 6593 log.cc:826] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:45.475216 6593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f: No bootstrap required, opened a new log
I20251027 14:51:45.476715 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:51:45.480834 6593 raft_consensus.cc:359] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223723076e8c4c7f95787345a1658b0f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44299 } }
I20251027 14:51:45.481158 6593 raft_consensus.cc:385] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:45.481245 6593 raft_consensus.cc:740] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 223723076e8c4c7f95787345a1658b0f, State: Initialized, Role: FOLLOWER
I20251027 14:51:45.481770 6593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223723076e8c4c7f95787345a1658b0f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44299 } }
I20251027 14:51:45.481953 6593 raft_consensus.cc:399] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:45.482060 6593 raft_consensus.cc:493] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:45.482195 6593 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:45.484685 6593 raft_consensus.cc:515] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223723076e8c4c7f95787345a1658b0f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44299 } }
I20251027 14:51:45.485162 6593 leader_election.cc:304] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 223723076e8c4c7f95787345a1658b0f; no voters:
I20251027 14:51:45.485713 6593 leader_election.cc:290] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:45.485816 6598 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:45.486681 6598 raft_consensus.cc:697] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [term 1 LEADER]: Becoming Leader. State: Replica: 223723076e8c4c7f95787345a1658b0f, State: Running, Role: LEADER
I20251027 14:51:45.487156 6598 consensus_queue.cc:237] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223723076e8c4c7f95787345a1658b0f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44299 } }
I20251027 14:51:45.487425 6593 sys_catalog.cc:565] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:45.489799 6600 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 223723076e8c4c7f95787345a1658b0f. Latest consensus state: current_term: 1 leader_uuid: "223723076e8c4c7f95787345a1658b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223723076e8c4c7f95787345a1658b0f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44299 } } }
I20251027 14:51:45.490159 6600 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:45.490597 6599 sys_catalog.cc:455] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "223723076e8c4c7f95787345a1658b0f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "223723076e8c4c7f95787345a1658b0f" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44299 } } }
I20251027 14:51:45.490857 6599 sys_catalog.cc:458] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:45.491291 6607 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:45.496559 6607 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:45.505618 6607 catalog_manager.cc:1357] Generated new cluster ID: 79c17fc5b517420f8fdc67c70320a323
I20251027 14:51:45.505730 6607 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:45.536342 6607 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:45.537550 6607 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:45.546809 6607 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 223723076e8c4c7f95787345a1658b0f: Generated new TSK 0
I20251027 14:51:45.547578 6607 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:51:45.647380 6596 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:45.567305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37158 (local address 127.0.104.62:44299)
1027 14:51:45.567487 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:45.567494 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:45.567947 (+ 453us) server_negotiation.cc:408] Connection header received
1027 14:51:45.568124 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:45.568131 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:45.568219 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:45.568398 (+ 179us) server_negotiation.cc:227] Negotiated authn=JWT
1027 14:51:45.569026 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:45.570447 (+ 1421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:45.571587 (+ 1140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:45.572138 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:45.572318 (+ 180us) server_negotiation.cc:366] Received JWT_EXCHANGE NegotiatePB request
1027 14:51:45.646644 (+ 74326us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1027 14:51:45.647099 (+ 455us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:37158: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"server-negotiator.queue_time_us":57}
W20251027 14:51:45.647432 6622 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:45.567260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44299 (local address 127.0.0.1:37158)
1027 14:51:45.567844 (+ 584us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:45.567859 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:45.567982 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:45.568367 (+ 385us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:45.568371 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:45.568413 (+ 42us) client_negotiation.cc:189] Negotiated authn=JWT
1027 14:51:45.568874 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:45.568885 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:45.570606 (+ 1721us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:45.570611 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:45.571462 (+ 851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:45.571473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:45.571592 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:45.571619 (+ 27us) client_negotiation.cc:252] Sending JWT_EXCHANGE NegotiatePB request
1027 14:51:45.646978 (+ 75359us) client_negotiation.cc:283] Received error response from server: Runtime error: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
1027 14:51:45.647190 (+ 212us) negotiation.cc:327] Negotiation complete: Runtime error: Client connection negotiation failed: client connection to 127.0.104.62:44299: FATAL_INVALID_JWT: Not authorized: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":363,"threads_started":1}
W20251027 14:51:45.647521 6596 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:37158: Failed to load JWKS: Error downloading JWKS from 'default.url': curl error: Could not resolve hostname: Could not resolve host: default.url
I20251027 14:51:45.648572 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6527
I20251027 14:51:45.649858 6588 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:45.747825 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6527
2025-10-27T14:51:45Z chronyd exiting
[ OK ] SecurityITest.TestJwtMiniClusterWithNonWorkingJWKS (768 ms)
[ RUN ] SecurityITest.TestWorldReadableKeytab
[ OK ] SecurityITest.TestWorldReadableKeytab (1407 ms)
[ RUN ] SecurityITest.TestWorldReadablePrivateKey
[ OK ] SecurityITest.TestWorldReadablePrivateKey (1401 ms)
[ RUN ] SecurityITest.TestCorruptKerberosCC
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:48 dist-test-slave-hxxt krb5kdc[6639](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:48 dist-test-slave-hxxt krb5kdc[6639](info): set up 2 sockets
Oct 27 14:51:48 dist-test-slave-hxxt krb5kdc[6639](info): commencing operation
krb5kdc: starting...
W20251027 14:51:50.440835 416 mini_kdc.cc:121] Time spent starting KDC: real 1.851s user 0.004s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:50 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:50.673823 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37253
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37253
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:50.984891 6655 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:50.985180 6655 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:50.985272 6655 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:50.993323 6655 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:50.993433 6655 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:50.993525 6655 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:50.993582 6655 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:50.993645 6655 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:51.004195 6655 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37253
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:51.005743 6655 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:51.007010 6655 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:51.015372 6661 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:51.015364 6663 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:51.015486 6660 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:51.016010 6655 server_base.cc:1047] running on GCE node
I20251027 14:51:51.016785 6655 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:51.018127 6655 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:51.019330 6655 hybrid_clock.cc:648] HybridClock initialized: now 1761576711019268 us; error 41 us; skew 500 ppm
I20251027 14:51:51.019649 6655 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:51 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:51.022612 6655 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:51:51.024052 6655 webserver.cc:492] Webserver started at http://127.0.104.62:34285/ using document root <none> and password file <none>
I20251027 14:51:51.024775 6655 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:51.024854 6655 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:51.025172 6655 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:51.027639 6655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "2876f013c4f54641a107862171638518"
format_stamp: "Formatted at 2025-10-27 14:51:51 on dist-test-slave-hxxt"
server_key: "8e7dd02f5689ab14779ca7ebbcf0868a"
server_key_iv: "cb817e820d6b97fd9396aabe87570ba6"
server_key_version: "encryptionkey@0"
I20251027 14:51:51.028332 6655 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "2876f013c4f54641a107862171638518"
format_stamp: "Formatted at 2025-10-27 14:51:51 on dist-test-slave-hxxt"
server_key: "8e7dd02f5689ab14779ca7ebbcf0868a"
server_key_iv: "cb817e820d6b97fd9396aabe87570ba6"
server_key_version: "encryptionkey@0"
I20251027 14:51:51.033469 6655 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:51:51.036670 6670 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:51.037864 6655 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:51:51.038013 6655 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "2876f013c4f54641a107862171638518"
format_stamp: "Formatted at 2025-10-27 14:51:51 on dist-test-slave-hxxt"
server_key: "8e7dd02f5689ab14779ca7ebbcf0868a"
server_key_iv: "cb817e820d6b97fd9396aabe87570ba6"
server_key_version: "encryptionkey@0"
I20251027 14:51:51.038189 6655 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:51.065169 6655 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:51.071456 6655 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:51.071705 6655 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:51.089638 6721 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37253 every 8 connection(s)
I20251027 14:51:51.089634 6655 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37253
I20251027 14:51:51.091084 6655 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:51.091490 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6655
I20251027 14:51:51.091686 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:51.092357 416 external_mini_cluster.cc:1468] Setting key a457fa057ca3813e5db68dc196daaca0
I20251027 14:51:51.095961 6722 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 27 14:51:51 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576710, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:51.107496 6722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518: Bootstrap starting.
I20251027 14:51:51.111326 6722 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:51.112618 6722 log.cc:826] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:51.113374 6725 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:51.094242 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44904 (local address 127.0.104.62:37253)
1027 14:51:51.094966 (+ 724us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:51.094981 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:51.095036 (+ 55us) server_negotiation.cc:408] Connection header received
1027 14:51:51.096808 (+ 1772us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:51.096837 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:51.097357 (+ 520us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:51.097861 (+ 504us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:51.099207 (+ 1346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:51.101274 (+ 2067us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:51.102353 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:51.102892 (+ 539us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:51.105993 (+ 3101us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:51.106088 (+ 95us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:51.106103 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:51.106136 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:51.109508 (+ 3372us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:51.110078 (+ 570us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:51.110087 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:51.110096 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:51.110212 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:51.110458 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:51.110463 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:51.110466 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:51.111028 (+ 562us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:51.111342 (+ 314us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:51.111632 (+ 290us) server_negotiation.cc:300] Negotiation successful
1027 14:51:51.112063 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":502,"thread_start_us":315,"threads_started":1}
I20251027 14:51:51.115727 6722 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518: No bootstrap required, opened a new log
I20251027 14:51:51.121037 6722 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2876f013c4f54641a107862171638518" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37253 } }
I20251027 14:51:51.121330 6722 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:51.121402 6722 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2876f013c4f54641a107862171638518, State: Initialized, Role: FOLLOWER
I20251027 14:51:51.121870 6722 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2876f013c4f54641a107862171638518" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37253 } }
I20251027 14:51:51.122032 6722 raft_consensus.cc:399] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:51.122117 6722 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:51.122227 6722 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:51.124541 6722 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2876f013c4f54641a107862171638518" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37253 } }
I20251027 14:51:51.125002 6722 leader_election.cc:304] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 2876f013c4f54641a107862171638518; no voters:
I20251027 14:51:51.125521 6722 leader_election.cc:290] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:51.125629 6727 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:51.126223 6727 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [term 1 LEADER]: Becoming Leader. State: Replica: 2876f013c4f54641a107862171638518, State: Running, Role: LEADER
I20251027 14:51:51.126672 6727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2876f013c4f54641a107862171638518" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37253 } }
I20251027 14:51:51.127300 6722 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:51.129491 6729 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2876f013c4f54641a107862171638518. Latest consensus state: current_term: 1 leader_uuid: "2876f013c4f54641a107862171638518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2876f013c4f54641a107862171638518" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37253 } } }
I20251027 14:51:51.129452 6728 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "2876f013c4f54641a107862171638518" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "2876f013c4f54641a107862171638518" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37253 } } }
I20251027 14:51:51.129812 6729 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:51.129819 6728 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:51.132350 6735 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:51.137785 6735 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:51.146689 6735 catalog_manager.cc:1357] Generated new cluster ID: 35a6f11a4ce942cb8c884c2404099a70
I20251027 14:51:51.146801 6735 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:51.156421 6735 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:51.157567 6735 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:51.171811 6735 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2876f013c4f54641a107862171638518: Generated new TSK 0
I20251027 14:51:51.172715 6735 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:51.269423 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37253
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:51.585804 6750 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:51.586122 6750 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:51.586272 6750 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:51.594342 6750 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:51.594472 6750 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:51.594653 6750 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:51.605484 6750 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:51.607052 6750 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:51.608402 6750 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:51.618037 6758 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:51.618100 6756 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:51.618068 6755 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:51.618760 6750 server_base.cc:1047] running on GCE node
I20251027 14:51:51.619323 6750 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:51.620160 6750 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:51.621382 6750 hybrid_clock.cc:648] HybridClock initialized: now 1761576711621294 us; error 65 us; skew 500 ppm
I20251027 14:51:51.621644 6750 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:51 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:51.624594 6750 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:51:51.626070 6750 webserver.cc:492] Webserver started at http://127.0.104.1:40259/ using document root <none> and password file <none>
I20251027 14:51:51.626767 6750 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:51.626857 6750 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:51.627190 6750 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:51.630060 6750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "90cba7bc4a984bccbd7c2791744c76e3"
format_stamp: "Formatted at 2025-10-27 14:51:51 on dist-test-slave-hxxt"
server_key: "f940bbcf6fed919631cda935d50b2230"
server_key_iv: "897ebeb0099ee12eeed42accfa1bc828"
server_key_version: "encryptionkey@0"
I20251027 14:51:51.630890 6750 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "90cba7bc4a984bccbd7c2791744c76e3"
format_stamp: "Formatted at 2025-10-27 14:51:51 on dist-test-slave-hxxt"
server_key: "f940bbcf6fed919631cda935d50b2230"
server_key_iv: "897ebeb0099ee12eeed42accfa1bc828"
server_key_version: "encryptionkey@0"
I20251027 14:51:51.636322 6750 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:51:51.639709 6765 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:51.641103 6750 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:51:51.641270 6750 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "90cba7bc4a984bccbd7c2791744c76e3"
format_stamp: "Formatted at 2025-10-27 14:51:51 on dist-test-slave-hxxt"
server_key: "f940bbcf6fed919631cda935d50b2230"
server_key_iv: "897ebeb0099ee12eeed42accfa1bc828"
server_key_version: "encryptionkey@0"
I20251027 14:51:51.641479 6750 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:51.653668 6750 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:51.657320 6750 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:51.657570 6750 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:51.658443 6750 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:51.659883 6750 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:51.659972 6750 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:51.660072 6750 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:51.660143 6750 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:51.688429 6750 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33941
I20251027 14:51:51.688486 6877 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33941 every 8 connection(s)
I20251027 14:51:51.689798 6750 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:51.690445 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6750
I20251027 14:51:51.690662 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:51.691270 416 external_mini_cluster.cc:1468] Setting key d36a91e545c7bbbc1be7831fff21081a
Oct 27 14:51:51 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576711, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:51.708539 6880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:51.692323 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:41003 (local address 127.0.104.62:37253)
1027 14:51:51.692748 (+ 425us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:51.692753 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:51.693975 (+ 1222us) server_negotiation.cc:408] Connection header received
1027 14:51:51.695487 (+ 1512us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:51.695493 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:51.695564 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:51.695679 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:51.697298 (+ 1619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:51.698491 (+ 1193us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:51.699994 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:51.700303 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:51.703363 (+ 3060us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:51.703390 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:51.703393 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:51.703421 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:51.705731 (+ 2310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:51.706373 (+ 642us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:51.706377 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:51.706379 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:51.706452 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:51.706849 (+ 397us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:51.706852 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:51.706854 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:51.707054 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:51.707162 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:51.708053 (+ 891us) server_negotiation.cc:300] Negotiation successful
1027 14:51:51.708248 (+ 195us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":222,"threads_started":1}
I20251027 14:51:51.709879 6881 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:51.693030 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37253 (local address 127.0.104.1:41003)
1027 14:51:51.693799 (+ 769us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:51.693844 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:51.695221 (+ 1377us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:51.695859 (+ 638us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:51.695867 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:51.696449 (+ 582us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:51.697131 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:51.697149 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:51.698675 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:51.698681 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:51.699841 (+ 1160us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:51.699857 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:51.700116 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:51.700847 (+ 731us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:51.700877 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:51.703183 (+ 2306us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:51.705858 (+ 2675us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:51.705867 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:51.705885 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:51.706256 (+ 371us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:51.706577 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:51.706583 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:51.706588 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:51.706742 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:51.707167 (+ 425us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:51.707174 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:51.707592 (+ 418us) client_negotiation.cc:769] Sending connection context
1027 14:51:51.707992 (+ 400us) client_negotiation.cc:240] Negotiation successful
1027 14:51:51.708515 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":360,"threads_started":1}
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:51:51.711306 6878 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37253
I20251027 14:51:51.711648 6878 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:51.712445 6878 heartbeater.cc:507] Master 127.0.104.62:37253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:51:51.714844 6687 ts_manager.cc:194] Registered new tserver with Master: 90cba7bc4a984bccbd7c2791744c76e3 (127.0.104.1:33941)
I20251027 14:51:51.716763 6687 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:41003
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:51.771606 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37253
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:52.092334 6886 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:52.092720 6886 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:52.092869 6886 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:52.106187 6886 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:52.106321 6886 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:52.106552 6886 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:52.119246 6886 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:52.120819 6886 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:52.122036 6886 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:52.131873 6891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:52.132120 6892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:52.132437 6894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:52.133193 6886 server_base.cc:1047] running on GCE node
I20251027 14:51:52.133795 6886 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:52.134789 6886 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:52.136034 6886 hybrid_clock.cc:648] HybridClock initialized: now 1761576712135959 us; error 46 us; skew 500 ppm
I20251027 14:51:52.136332 6886 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.139514 6886 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:51:52.141136 6886 webserver.cc:492] Webserver started at http://127.0.104.2:35547/ using document root <none> and password file <none>
I20251027 14:51:52.141891 6886 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:52.141973 6886 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:52.142372 6886 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:52.145136 6886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "ab53f1453cae4db18d520138167097d0"
format_stamp: "Formatted at 2025-10-27 14:51:52 on dist-test-slave-hxxt"
server_key: "a6197734f96514087cec757300884f94"
server_key_iv: "f9fedad99ecac1553a75a35e5f3f5153"
server_key_version: "encryptionkey@0"
I20251027 14:51:52.146217 6886 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "ab53f1453cae4db18d520138167097d0"
format_stamp: "Formatted at 2025-10-27 14:51:52 on dist-test-slave-hxxt"
server_key: "a6197734f96514087cec757300884f94"
server_key_iv: "f9fedad99ecac1553a75a35e5f3f5153"
server_key_version: "encryptionkey@0"
I20251027 14:51:52.152094 6886 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:51:52.155764 6901 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:52.157297 6886 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:51:52.157456 6886 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "ab53f1453cae4db18d520138167097d0"
format_stamp: "Formatted at 2025-10-27 14:51:52 on dist-test-slave-hxxt"
server_key: "a6197734f96514087cec757300884f94"
server_key_iv: "f9fedad99ecac1553a75a35e5f3f5153"
server_key_version: "encryptionkey@0"
I20251027 14:51:52.157677 6886 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:52.178634 6886 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:52.182879 6886 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:52.183135 6886 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:52.184034 6886 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:52.185609 6886 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:52.185698 6886 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:52.185804 6886 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:52.185868 6886 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:52.217471 6886 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42369
I20251027 14:51:52.217510 7013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42369 every 8 connection(s)
I20251027 14:51:52.218763 6886 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:52.220719 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 6886
I20251027 14:51:52.220880 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:52.221241 416 external_mini_cluster.cc:1468] Setting key 8c335d1ed34f3e2256c65f592aa265be
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:52.237380 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.221440 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:55093 (local address 127.0.104.62:37253)
1027 14:51:52.221865 (+ 425us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.221869 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.223035 (+ 1166us) server_negotiation.cc:408] Connection header received
1027 14:51:52.224585 (+ 1550us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.224591 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.224674 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.224772 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.226429 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.227331 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.228435 (+ 1104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.228911 (+ 476us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.231865 (+ 2954us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.231903 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.231927 (+ 24us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.231963 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.234316 (+ 2353us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.235145 (+ 829us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.235150 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.235152 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.235219 (+ 67us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.235573 (+ 354us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.235578 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.235581 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.235828 (+ 247us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.235934 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.236897 (+ 963us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.237089 (+ 192us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":224,"threads_started":1}
I20251027 14:51:52.238713 7017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.222106 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37253 (local address 127.0.104.2:55093)
1027 14:51:52.222874 (+ 768us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:52.222928 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:52.224275 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:52.225019 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:52.225027 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:52.225584 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:52.226247 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:52.226266 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.227488 (+ 1222us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.227492 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:52.228270 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:52.228281 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.228561 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.229395 (+ 834us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:52.229424 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:52.231675 (+ 2251us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:52.234459 (+ 2784us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:52.234468 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:52.234487 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:52.235017 (+ 530us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:52.235315 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:52.235319 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:52.235323 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:52.235468 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:52.235945 (+ 477us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:52.235954 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:52.236411 (+ 457us) client_negotiation.cc:769] Sending connection context
1027 14:51:52.236808 (+ 397us) client_negotiation.cc:240] Negotiation successful
1027 14:51:52.237330 (+ 522us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":345,"threads_started":1}
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:51:52.240173 7014 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37253
I20251027 14:51:52.240567 7014 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:52.241408 7014 heartbeater.cc:507] Master 127.0.104.62:37253 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
I20251027 14:51:52.242972 6687 ts_manager.cc:194] Registered new tserver with Master: ab53f1453cae4db18d520138167097d0 (127.0.104.2:42369)
I20251027 14:51:52.243800 6687 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:55093
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:51:52.302898 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37253
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:36415
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:52.616298 7022 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:52.616657 7022 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:52.616792 7022 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:52.624895 7022 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:52.625010 7022 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:52.625200 7022 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:52.635578 7022 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:36415
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37253
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:52.637228 7022 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:52.638537 7022 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:52.647964 7028 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:52.647987 7027 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:52.648216 7030 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:52.648655 7022 server_base.cc:1047] running on GCE node
I20251027 14:51:52.649205 7022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:52.650060 7022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:52.651278 7022 hybrid_clock.cc:648] HybridClock initialized: now 1761576712651213 us; error 37 us; skew 500 ppm
I20251027 14:51:52.651551 7022 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.654650 7022 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:51:52.656277 7022 webserver.cc:492] Webserver started at http://127.0.104.3:33071/ using document root <none> and password file <none>
I20251027 14:51:52.656988 7022 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:52.657075 7022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:52.657446 7022 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:52.659929 7022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "39c1435b325349418f83870e20eb857e"
format_stamp: "Formatted at 2025-10-27 14:51:52 on dist-test-slave-hxxt"
server_key: "4f752821f16208281c14fe450ffad586"
server_key_iv: "3a881c797753cd85544031b577f47042"
server_key_version: "encryptionkey@0"
I20251027 14:51:52.660712 7022 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "39c1435b325349418f83870e20eb857e"
format_stamp: "Formatted at 2025-10-27 14:51:52 on dist-test-slave-hxxt"
server_key: "4f752821f16208281c14fe450ffad586"
server_key_iv: "3a881c797753cd85544031b577f47042"
server_key_version: "encryptionkey@0"
I20251027 14:51:52.665954 7022 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20251027 14:51:52.669305 7037 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:52.670622 7022 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:51:52.670778 7022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "39c1435b325349418f83870e20eb857e"
format_stamp: "Formatted at 2025-10-27 14:51:52 on dist-test-slave-hxxt"
server_key: "4f752821f16208281c14fe450ffad586"
server_key_iv: "3a881c797753cd85544031b577f47042"
server_key_version: "encryptionkey@0"
I20251027 14:51:52.670979 7022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:52.683945 7022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:52.687583 7022 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:52.687834 7022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:52.688820 7022 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:52.690347 7022 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:52.690440 7022 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:52.690541 7022 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:52.690608 7022 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:52.719748 6878 heartbeater.cc:499] Master 127.0.104.62:37253 was elected leader, sending a full tablet report...
I20251027 14:51:52.721148 7022 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46593
I20251027 14:51:52.721197 7149 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46593 every 8 connection(s)
I20251027 14:51:52.722570 7022 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:52.731159 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7022
I20251027 14:51:52.731336 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:52.731721 416 external_mini_cluster.cc:1468] Setting key 655f020bdb482202363ed46f25d0ffac
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:51:52.740490 7016 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.725090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:43233 (local address 127.0.104.62:37253)
1027 14:51:52.725273 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.725278 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.726523 (+ 1245us) server_negotiation.cc:408] Connection header received
1027 14:51:52.728015 (+ 1492us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.728022 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.728106 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.728194 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.730044 (+ 1850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.730991 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.732129 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.732672 (+ 543us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.735624 (+ 2952us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.735655 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.735659 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.735687 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.737985 (+ 2298us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.738606 (+ 621us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.738609 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.738611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.738668 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.738906 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.738909 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.738911 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.739108 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.739203 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.739898 (+ 695us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.740148 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251027 14:51:52.741417 7152 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.725640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37253 (local address 127.0.104.3:43233)
1027 14:51:52.726368 (+ 728us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:52.726407 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:52.727742 (+ 1335us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:52.728484 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:52.728493 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:52.729079 (+ 586us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:52.729771 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:52.729790 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.731145 (+ 1355us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.731150 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:52.732000 (+ 850us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:52.732013 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.732271 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.733199 (+ 928us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:52.733232 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:52.735442 (+ 2210us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:52.738105 (+ 2663us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:52.738112 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:52.738145 (+ 33us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:52.738513 (+ 368us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:52.738738 (+ 225us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:52.738741 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:52.738743 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:52.738838 (+ 95us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:52.739199 (+ 361us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:52.739204 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:52.739479 (+ 275us) client_negotiation.cc:769] Sending connection context
1027 14:51:52.739795 (+ 316us) client_negotiation.cc:240] Negotiation successful
1027 14:51:52.740258 (+ 463us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"thread_start_us":311,"threads_started":1}
I20251027 14:51:52.742606 7150 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37253
I20251027 14:51:52.742923 7150 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:52.743706 7150 heartbeater.cc:507] Master 127.0.104.62:37253 requested a full tablet report, sending...
I20251027 14:51:52.745019 6687 ts_manager.cc:194] Registered new tserver with Master: 39c1435b325349418f83870e20eb857e (127.0.104.3:46593)
I20251027 14:51:52.746006 6687 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:43233
I20251027 14:51:52.746917 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal test-admin with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/test-admin.keytab.
Entry for principal test-admin with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/test-admin.keytab.
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.770828 416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
W20251027 14:51:52.775691 7157 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:52.773383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33620 (local address 127.0.104.1:33941)
1027 14:51:52.773976 (+ 593us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.773983 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.774009 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:51:52.774071 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.774077 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.774281 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.774460 (+ 179us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.775359 (+ 899us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33620: BlockingRecv error: recv got EOF from 127.0.0.1:33620 (error 108)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":324,"threads_started":1}
W20251027 14:51:52.775929 416 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc' of type FILE
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.777346 416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.778563 416 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:51:52.792960 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.780206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33630 (local address 127.0.104.1:33941)
1027 14:51:52.780395 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.780401 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.780466 (+ 65us) server_negotiation.cc:408] Connection header received
1027 14:51:52.780642 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.780649 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.780704 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.780780 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.782005 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.783243 (+ 1238us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.784413 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.784848 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.787582 (+ 2734us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.787622 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.787637 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.787668 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.790496 (+ 2828us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.791151 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.791159 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.791161 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.791239 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.791536 (+ 297us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.791541 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.791544 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.791821 (+ 277us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.792128 (+ 307us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.792436 (+ 308us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.792696 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
W20251027 14:51:52.798708 7157 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:52.797053 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33636 (local address 127.0.104.1:33941)
1027 14:51:52.797238 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.797243 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.797506 (+ 263us) server_negotiation.cc:408] Connection header received
1027 14:51:52.797659 (+ 153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.797663 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.797718 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.797802 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.798479 (+ 677us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33636: BlockingRecv error: recv got EOF from 127.0.0.1:33636 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20251027 14:51:52.799090 416 init.cc:244] Error while opening credential cache '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc' of type FILE: Runtime error: End of credential cache reached (filename: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc)
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.800571 416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.801859 416 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:51:52.814936 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.803195 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33638 (local address 127.0.104.1:33941)
1027 14:51:52.803404 (+ 209us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.803411 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.803683 (+ 272us) server_negotiation.cc:408] Connection header received
1027 14:51:52.803847 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.803853 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.803913 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.803991 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.805216 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.806108 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.807195 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.807609 (+ 414us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.810357 (+ 2748us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.810386 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.810389 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.810421 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.812870 (+ 2449us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.813435 (+ 565us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.813441 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.813444 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.813528 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.813785 (+ 257us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.813790 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.813793 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.813994 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.814244 (+ 250us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.814483 (+ 239us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.814696 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20251027 14:51:52.818895 7157 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:51:52.817386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.104.1:33941)
1027 14:51:52.817578 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.817583 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.817815 (+ 232us) server_negotiation.cc:408] Connection header received
1027 14:51:52.817945 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.817948 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.818002 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.818070 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.818698 (+ 628us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33646: BlockingRecv error: recv got EOF from 127.0.0.1:33646 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
W20251027 14:51:52.819067 416 init.cc:300] Could not find kerberos principal in credential cache '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestCorruptKerberosCC.1761576633729408-416-0/krb5kdc/krb5cc' of type FILE
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.820369 416 init.cc:377] Logged in from keytab as test-admin@KRBTEST.COM (short username test-admin)
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:52.821540 416 init.cc:295] Successfully reacquired a new kerberos TGT
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:51:52.834318 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.822821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33650 (local address 127.0.104.1:33941)
1027 14:51:52.822972 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.822977 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.823241 (+ 264us) server_negotiation.cc:408] Connection header received
1027 14:51:52.823387 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.823393 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.823450 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.823520 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.824639 (+ 1119us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.825454 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.826487 (+ 1033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.826910 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.829559 (+ 2649us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.829584 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.829588 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.829616 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.832126 (+ 2510us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.832793 (+ 667us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.832799 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.832801 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.832867 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.833141 (+ 274us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.833144 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.833146 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.833349 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.833572 (+ 223us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.833846 (+ 274us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.834063 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251027 14:51:52.835784 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6750
I20251027 14:51:52.846333 7157 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.836514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33664 (local address 127.0.104.1:33941)
1027 14:51:52.836673 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.836677 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.836860 (+ 183us) server_negotiation.cc:408] Connection header received
1027 14:51:52.836992 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.836996 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.837045 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.837119 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.838194 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.839033 (+ 839us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.840009 (+ 976us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.840298 (+ 289us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.841664 (+ 1366us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.841693 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.841696 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.841718 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.844105 (+ 2387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.844690 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.844694 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.844696 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.844766 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.845062 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.845065 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.845067 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.845276 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.845455 (+ 179us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.845777 (+ 322us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.846004 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20251027 14:51:52.846982 6873 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:52.930081 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6750
I20251027 14:51:52.943178 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6886
Oct 27 14:51:52 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:51:52.956888 7170 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:52.944058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35876 (local address 127.0.104.2:42369)
1027 14:51:52.944549 (+ 491us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:52.944556 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:52.944580 (+ 24us) server_negotiation.cc:408] Connection header received
1027 14:51:52.944638 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:52.944644 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:52.944846 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:52.944993 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:52.946080 (+ 1087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.947199 (+ 1119us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:52.948335 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:52.948810 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:52.951666 (+ 2856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:52.951740 (+ 74us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:52.951756 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:52.951786 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:52.954407 (+ 2621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.955009 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.955017 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.955020 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.955091 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:52.955381 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:52.955385 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:52.955387 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:52.955671 (+ 284us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:52.955953 (+ 282us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:52.956214 (+ 261us) server_negotiation.cc:300] Negotiation successful
1027 14:51:52.956606 (+ 392us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":246,"threads_started":1}
I20251027 14:51:52.958056 7009 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:53.039004 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6886
I20251027 14:51:53.051590 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7022
Oct 27 14:51:53 dist-test-slave-hxxt krb5kdc[6639](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576712, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:51:53.065979 7172 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:53.052501 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37824 (local address 127.0.104.3:46593)
1027 14:51:53.053152 (+ 651us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:53.053160 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:53.053188 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:51:53.053249 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:53.053256 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:53.053461 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:53.053614 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:53.054731 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:53.055950 (+ 1219us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:53.057296 (+ 1346us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:53.057797 (+ 501us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:53.060783 (+ 2986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:53.060843 (+ 60us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:53.060857 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:53.060889 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:53.063656 (+ 2767us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:53.064300 (+ 644us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:53.064308 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:53.064311 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:53.064492 (+ 181us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:53.064748 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:53.064751 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:53.064753 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:53.065003 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:53.065222 (+ 219us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:53.065462 (+ 240us) server_negotiation.cc:300] Negotiation successful
1027 14:51:53.065691 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":489,"thread_start_us":363,"threads_started":1}
I20251027 14:51:53.067202 7145 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:53.148761 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7022
I20251027 14:51:53.161131 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 6655
I20251027 14:51:53.162266 6717 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:51:53.262544 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 6655
2025-10-27T14:51:53Z chronyd exiting
[ OK ] SecurityITest.TestCorruptKerberosCC (4709 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipalMultipleMaster
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:51:53 dist-test-slave-hxxt krb5kdc[7177](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:51:53 dist-test-slave-hxxt krb5kdc[7177](info): set up 2 sockets
Oct 27 14:51:53 dist-test-slave-hxxt krb5kdc[7177](info): commencing operation
krb5kdc: starting...
W20251027 14:51:55.128859 416 mini_kdc.cc:121] Time spent starting KDC: real 1.829s user 0.002s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:51:55 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:51:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:51:55Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:51:55.367503 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44599
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44599
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:55.679863 7193 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:55.680159 7193 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:55.680255 7193 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:55.688448 7193 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:55.688550 7193 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:55.688607 7193 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:55.688653 7193 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:55.688712 7193 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:55.699218 7193 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44599
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44599
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:55.700713 7193 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:55.702015 7193 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:55.709455 7199 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:55.709481 7198 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:55.710121 7201 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:55.710546 7193 server_base.cc:1047] running on GCE node
I20251027 14:51:55.711305 7193 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:55.712721 7193 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:55.713945 7193 hybrid_clock.cc:648] HybridClock initialized: now 1761576715713884 us; error 44 us; skew 500 ppm
I20251027 14:51:55.714288 7193 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:55 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:55.717245 7193 init.cc:377] Logged in from keytab as oryx/127.0.104.62@KRBTEST.COM (short username oryx)
I20251027 14:51:55.718745 7193 webserver.cc:492] Webserver started at http://127.0.104.62:33671/ using document root <none> and password file <none>
I20251027 14:51:55.719430 7193 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:55.719506 7193 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:55.719805 7193 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:55.722348 7193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "ebe7c276923347a3ae5401422b9dab66"
format_stamp: "Formatted at 2025-10-27 14:51:55 on dist-test-slave-hxxt"
server_key: "35ceda8d568280f7656e4d987a585752"
server_key_iv: "8cf319c63e9527f13103223ecbb4d31f"
server_key_version: "encryptionkey@0"
I20251027 14:51:55.723044 7193 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "ebe7c276923347a3ae5401422b9dab66"
format_stamp: "Formatted at 2025-10-27 14:51:55 on dist-test-slave-hxxt"
server_key: "35ceda8d568280f7656e4d987a585752"
server_key_iv: "8cf319c63e9527f13103223ecbb4d31f"
server_key_version: "encryptionkey@0"
I20251027 14:51:55.728266 7193 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251027 14:51:55.731604 7208 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:55.733037 7193 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:51:55.733183 7193 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "ebe7c276923347a3ae5401422b9dab66"
format_stamp: "Formatted at 2025-10-27 14:51:55 on dist-test-slave-hxxt"
server_key: "35ceda8d568280f7656e4d987a585752"
server_key_iv: "8cf319c63e9527f13103223ecbb4d31f"
server_key_version: "encryptionkey@0"
I20251027 14:51:55.733362 7193 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:55.752104 7193 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:55.755926 7193 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:55.756150 7193 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:55.774037 7193 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44599
I20251027 14:51:55.774056 7259 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44599 every 8 connection(s)
I20251027 14:51:55.775437 7193 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:51:55.779520 7260 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:55.785691 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7193
I20251027 14:51:55.785938 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:51:55.786492 416 external_mini_cluster.cc:1468] Setting key 1fe4f0a77ca8aadd4f4467b250727d78
I20251027 14:51:55.790040 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Bootstrap starting.
I20251027 14:51:55.793740 7260 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:55.794790 7260 log.cc:826] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:55.798017 7260 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: No bootstrap required, opened a new log
Oct 27 14:51:55 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:51:55.803390 7260 raft_consensus.cc:359] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } }
I20251027 14:51:55.803725 7260 raft_consensus.cc:385] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:55.803830 7260 raft_consensus.cc:740] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ebe7c276923347a3ae5401422b9dab66, State: Initialized, Role: FOLLOWER
I20251027 14:51:55.804522 7260 consensus_queue.cc:260] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } }
I20251027 14:51:55.804716 7260 raft_consensus.cc:399] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:51:55.804868 7260 raft_consensus.cc:493] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:51:55.805023 7260 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:55.806890 7263 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:55.788530 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47538 (local address 127.0.104.62:44599)
1027 14:51:55.789415 (+ 885us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:55.789427 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:55.789466 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:51:55.790708 (+ 1242us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:55.790730 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:55.791163 (+ 433us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:55.791553 (+ 390us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:55.793478 (+ 1925us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:55.794856 (+ 1378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:55.796081 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:55.796565 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:55.800163 (+ 3598us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:55.800220 (+ 57us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:55.800234 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:55.800266 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:55.803091 (+ 2825us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:55.803654 (+ 563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:55.803661 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:55.803668 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:55.803793 (+ 125us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:55.804144 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:55.804150 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:55.804153 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:55.804684 (+ 531us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:55.804980 (+ 296us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:55.805414 (+ 434us) server_negotiation.cc:300] Negotiation successful
1027 14:51:55.805885 (+ 471us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":685,"thread_start_us":501,"threads_started":1}
I20251027 14:51:55.807512 7260 raft_consensus.cc:515] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } }
I20251027 14:51:55.808068 7260 leader_election.cc:304] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ebe7c276923347a3ae5401422b9dab66; no voters:
I20251027 14:51:55.808847 7260 leader_election.cc:290] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:51:55.808964 7265 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:51:55.810349 7265 raft_consensus.cc:697] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 LEADER]: Becoming Leader. State: Replica: ebe7c276923347a3ae5401422b9dab66, State: Running, Role: LEADER
I20251027 14:51:55.810802 7265 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } }
I20251027 14:51:55.811074 7260 sys_catalog.cc:565] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:55.814288 7267 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe7c276923347a3ae5401422b9dab66. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } }
I20251027 14:51:55.814769 7267 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:55.814779 7266 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } }
I20251027 14:51:55.815110 7266 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:55.815713 7275 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:51:55.820686 7275 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:51:55.831007 7275 catalog_manager.cc:1357] Generated new cluster ID: 1d1d091730ea451bb60f617614e64065
I20251027 14:51:55.831210 7275 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:51:55.849210 7275 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:51:55.850576 7275 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:51:55.859539 7275 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Generated new TSK 0
I20251027 14:51:55.860353 7275 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:51:55.957437 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44599
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:56.276790 7288 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:56.277096 7288 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:56.277223 7288 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:56.285112 7288 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:56.285207 7288 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:56.285384 7288 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:51:56.295650 7288 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:56.297252 7288 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:56.298520 7288 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:56.307227 7293 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:56.307163 7294 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:56.308341 7296 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:56.308341 7288 server_base.cc:1047] running on GCE node
I20251027 14:51:56.309110 7288 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:56.309947 7288 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:56.311153 7288 hybrid_clock.cc:648] HybridClock initialized: now 1761576716311095 us; error 32 us; skew 500 ppm
I20251027 14:51:56.311416 7288 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:56 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:56.314385 7288 init.cc:377] Logged in from keytab as oryx/127.0.104.1@KRBTEST.COM (short username oryx)
I20251027 14:51:56.316000 7288 webserver.cc:492] Webserver started at http://127.0.104.1:38495/ using document root <none> and password file <none>
I20251027 14:51:56.316741 7288 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:56.316831 7288 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:56.317178 7288 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:56.319614 7288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "45af0ea8283a4637babe28d2c6441f83"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "4fa7be8cde7bdaa929b7783fe5a1ebbd"
server_key_iv: "8264a858a9c185f5953ce03a7f6704b6"
server_key_version: "encryptionkey@0"
I20251027 14:51:56.320417 7288 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "45af0ea8283a4637babe28d2c6441f83"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "4fa7be8cde7bdaa929b7783fe5a1ebbd"
server_key_iv: "8264a858a9c185f5953ce03a7f6704b6"
server_key_version: "encryptionkey@0"
I20251027 14:51:56.325371 7288 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:51:56.328503 7303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:56.329766 7288 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:51:56.329921 7288 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "45af0ea8283a4637babe28d2c6441f83"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "4fa7be8cde7bdaa929b7783fe5a1ebbd"
server_key_iv: "8264a858a9c185f5953ce03a7f6704b6"
server_key_version: "encryptionkey@0"
I20251027 14:51:56.330122 7288 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:56.353796 7288 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:56.363584 7288 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:56.363865 7288 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:56.364878 7288 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:56.366343 7288 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:56.366436 7288 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:56.366535 7288 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:56.366606 7288 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:56.396298 7288 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35535
I20251027 14:51:56.396342 7415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35535 every 8 connection(s)
I20251027 14:51:56.397611 7288 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:51:56.399365 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7288
I20251027 14:51:56.399573 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:51:56.400244 416 external_mini_cluster.cc:1468] Setting key 658d94a6f451f083039d5215cf8bc197
Oct 27 14:51:56 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:51:56.420325 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:56.400774 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:50455 (local address 127.0.104.62:44599)
1027 14:51:56.405071 (+ 4297us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:56.405078 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:56.405833 (+ 755us) server_negotiation.cc:408] Connection header received
1027 14:51:56.407337 (+ 1504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:56.407343 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:56.407423 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:56.407529 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:56.409306 (+ 1777us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.410541 (+ 1235us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:56.411669 (+ 1128us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.412098 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:56.414614 (+ 2516us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:56.414648 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:56.414652 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:56.414686 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:56.417568 (+ 2882us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:56.418259 (+ 691us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:56.418264 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:56.418266 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:56.418343 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:56.418660 (+ 317us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:56.418665 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:56.418667 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:56.418897 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:56.419036 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:56.419737 (+ 701us) server_negotiation.cc:300] Negotiation successful
1027 14:51:56.419984 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4165,"thread_start_us":4067,"threads_started":1}
I20251027 14:51:56.421278 7419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:56.401268 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:50455)
1027 14:51:56.405684 (+ 4416us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:56.405725 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:56.407055 (+ 1330us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:56.407789 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:56.407800 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:56.408367 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:56.409126 (+ 759us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:56.409145 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.410711 (+ 1566us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:56.410715 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:56.411532 (+ 817us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:56.411544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.411755 (+ 211us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:56.412443 (+ 688us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:56.412468 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:56.414435 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:56.417708 (+ 3273us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:56.417715 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:56.417730 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:56.418131 (+ 401us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:56.418440 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:56.418444 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:56.418446 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:56.418554 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:56.419015 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:56.419023 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:56.419342 (+ 319us) client_negotiation.cc:769] Sending connection context
1027 14:51:56.419626 (+ 284us) client_negotiation.cc:240] Negotiation successful
1027 14:51:56.420118 (+ 492us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4053,"thread_start_us":3833,"threads_started":1}
WARNING: no policy specified for oryx/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:51:56.422762 7416 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44599
I20251027 14:51:56.423092 7416 heartbeater.cc:461] Registering TS with master...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.2@KRBTEST.COM" created.
I20251027 14:51:56.424000 7416 heartbeater.cc:507] Master 127.0.104.62:44599 requested a full tablet report, sending...
I20251027 14:51:56.426163 7225 ts_manager.cc:194] Registered new tserver with Master: 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535)
I20251027 14:51:56.427646 7225 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:50455
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:51:56.483239 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44599
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:56.797590 7424 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:56.797919 7424 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:56.798051 7424 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:56.806102 7424 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:56.806205 7424 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:56.806385 7424 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:51:56.817055 7424 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:56.818584 7424 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:56.819918 7424 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:56.829280 7429 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:56.829322 7432 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:56.829250 7430 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:56.830474 7424 server_base.cc:1047] running on GCE node
I20251027 14:51:56.831097 7424 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:56.832020 7424 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:56.833276 7424 hybrid_clock.cc:648] HybridClock initialized: now 1761576716833196 us; error 49 us; skew 500 ppm
I20251027 14:51:56.833580 7424 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:56 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:56.836853 7424 init.cc:377] Logged in from keytab as oryx/127.0.104.2@KRBTEST.COM (short username oryx)
I20251027 14:51:56.838466 7424 webserver.cc:492] Webserver started at http://127.0.104.2:36459/ using document root <none> and password file <none>
I20251027 14:51:56.839179 7424 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:56.839262 7424 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:56.839558 7424 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:56.842057 7424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "69e31b5ad78c4358b932f61f8dc1d662"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "13ec69f77de5795402e4a2eb2a2dd1b2"
server_key_iv: "dbf57f4d07308a55e523a888469a39e3"
server_key_version: "encryptionkey@0"
I20251027 14:51:56.842763 7424 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "69e31b5ad78c4358b932f61f8dc1d662"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "13ec69f77de5795402e4a2eb2a2dd1b2"
server_key_iv: "dbf57f4d07308a55e523a888469a39e3"
server_key_version: "encryptionkey@0"
I20251027 14:51:56.849526 7424 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20251027 14:51:56.853972 7439 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:56.855432 7424 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.004s
I20251027 14:51:56.855572 7424 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "69e31b5ad78c4358b932f61f8dc1d662"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "13ec69f77de5795402e4a2eb2a2dd1b2"
server_key_iv: "dbf57f4d07308a55e523a888469a39e3"
server_key_version: "encryptionkey@0"
I20251027 14:51:56.855774 7424 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:56.868909 7424 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:56.872942 7424 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:56.873160 7424 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:56.874029 7424 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:56.875422 7424 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:56.875504 7424 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:56.875579 7424 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:56.875633 7424 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:56.905416 7424 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:46289
I20251027 14:51:56.905484 7551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:46289 every 8 connection(s)
I20251027 14:51:56.906740 7424 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:51:56.912302 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7424
I20251027 14:51:56.912531 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:51:56.912971 416 external_mini_cluster.cc:1468] Setting key 39c643dd57cf537e28ce88c10007fb98
Oct 27 14:51:56 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576716, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:51:56.925518 7418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:56.909284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:53371 (local address 127.0.104.62:44599)
1027 14:51:56.909483 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:56.909490 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:56.910770 (+ 1280us) server_negotiation.cc:408] Connection header received
1027 14:51:56.912366 (+ 1596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:56.912387 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:56.912471 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:56.912563 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:56.914624 (+ 2061us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.915528 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:56.916992 (+ 1464us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.917280 (+ 288us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:56.920513 (+ 3233us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:56.920546 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:56.920550 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:56.920578 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:56.922912 (+ 2334us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:56.923544 (+ 632us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:56.923549 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:56.923553 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:56.923642 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:56.923906 (+ 264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:56.923911 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:56.923914 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:56.924181 (+ 267us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:56.924314 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:56.924992 (+ 678us) server_negotiation.cc:300] Negotiation successful
1027 14:51:56.925226 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:51:56.926334 7554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:56.909876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:53371)
1027 14:51:56.910609 (+ 733us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:56.910646 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:56.912053 (+ 1407us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:56.912837 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:56.912850 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:56.913626 (+ 776us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:56.914436 (+ 810us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:56.914457 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.915677 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:56.915682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:56.916868 (+ 1186us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:56.916882 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:56.917131 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:56.917811 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:56.917840 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:56.920300 (+ 2460us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:56.923037 (+ 2737us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:56.923044 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:56.923058 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:56.923438 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:56.923727 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:56.923730 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:56.923733 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:56.923833 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:56.924274 (+ 441us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:56.924280 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:56.924608 (+ 328us) client_negotiation.cc:769] Sending connection context
1027 14:51:56.924906 (+ 298us) client_negotiation.cc:240] Negotiation successful
1027 14:51:56.925250 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":321,"threads_started":1}
I20251027 14:51:56.928006 7552 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44599
I20251027 14:51:56.928417 7552 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:56.929432 7552 heartbeater.cc:507] Master 127.0.104.62:44599 requested a full tablet report, sending...
I20251027 14:51:56.931228 7225 ts_manager.cc:194] Registered new tserver with Master: 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289)
I20251027 14:51:56.932035 7225 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:53371
WARNING: no policy specified for oryx/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:51:56.994829 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:44599
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:57.323637 7559 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:57.323956 7559 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:57.324087 7559 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:57.332098 7559 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:57.332213 7559 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:57.332423 7559 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:51:57.342859 7559 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44599
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:57.344452 7559 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:57.345850 7559 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:57.355033 7564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:57.355062 7565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:57.355516 7567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:57.356112 7559 server_base.cc:1047] running on GCE node
I20251027 14:51:57.356740 7559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:57.357617 7559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:57.358851 7559 hybrid_clock.cc:648] HybridClock initialized: now 1761576717358792 us; error 34 us; skew 500 ppm
I20251027 14:51:57.359138 7559 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:57 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:57.362346 7559 init.cc:377] Logged in from keytab as oryx/127.0.104.3@KRBTEST.COM (short username oryx)
I20251027 14:51:57.363965 7559 webserver.cc:492] Webserver started at http://127.0.104.3:45311/ using document root <none> and password file <none>
I20251027 14:51:57.364732 7559 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:57.364820 7559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:57.365196 7559 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:57.367733 7559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "af3b3966dd764458af1f02e364788f7a"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "9241d96096bdb631cfd44b92139b6410"
server_key_iv: "8e7e06292a909bc115a7e3774ffc502a"
server_key_version: "encryptionkey@0"
I20251027 14:51:57.368507 7559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "af3b3966dd764458af1f02e364788f7a"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "9241d96096bdb631cfd44b92139b6410"
server_key_iv: "8e7e06292a909bc115a7e3774ffc502a"
server_key_version: "encryptionkey@0"
I20251027 14:51:57.373595 7559 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:51:57.376756 7574 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:57.378015 7559 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:51:57.378160 7559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "af3b3966dd764458af1f02e364788f7a"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "9241d96096bdb631cfd44b92139b6410"
server_key_iv: "8e7e06292a909bc115a7e3774ffc502a"
server_key_version: "encryptionkey@0"
I20251027 14:51:57.378358 7559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:57.398455 7559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:57.403475 7559 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:57.403772 7559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:57.404924 7559 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:51:57.406793 7559 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:51:57.406903 7559 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:57.407017 7559 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:51:57.407090 7559 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:57.430958 7416 heartbeater.cc:499] Master 127.0.104.62:44599 was elected leader, sending a full tablet report...
I20251027 14:51:57.445117 7559 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45563
I20251027 14:51:57.445204 7686 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45563 every 8 connection(s)
I20251027 14:51:57.446475 7559 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:51:57.454552 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7559
I20251027 14:51:57.454739 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:51:57.455094 416 external_mini_cluster.cc:1468] Setting key b86bf34abc979c1be5fe61b839b14e3a
Oct 27 14:51:57 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:51:57.466595 7689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:57.449436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:57599 (local address 127.0.104.62:44599)
1027 14:51:57.449931 (+ 495us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:57.449938 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:57.450952 (+ 1014us) server_negotiation.cc:408] Connection header received
1027 14:51:57.453026 (+ 2074us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:57.453031 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:57.453097 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:57.453196 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:57.455297 (+ 2101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:57.456520 (+ 1223us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:57.458032 (+ 1512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:57.458358 (+ 326us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:57.461499 (+ 3141us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:57.461531 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:57.461535 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:57.461564 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:57.463942 (+ 2378us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:57.464600 (+ 658us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:57.464606 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:57.464609 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:57.464684 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:57.464988 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:57.464992 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:57.464994 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:57.465216 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:57.465337 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:57.466015 (+ 678us) server_negotiation.cc:300] Negotiation successful
1027 14:51:57.466247 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":235,"threads_started":1}
I20251027 14:51:57.467476 7690 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:57.449994 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:57599)
1027 14:51:57.450761 (+ 767us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:57.450813 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:57.452688 (+ 1875us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:57.453443 (+ 755us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:57.453454 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:57.454212 (+ 758us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:57.455102 (+ 890us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:57.455128 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:57.456680 (+ 1552us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:57.456686 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:57.457885 (+ 1199us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:57.457899 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:57.458162 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:57.458824 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:57.458856 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:57.461321 (+ 2465us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:57.464068 (+ 2747us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:57.464077 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:57.464095 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:57.464502 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:57.464782 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:57.464787 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:57.464791 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:57.464890 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:57.465333 (+ 443us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:57.465341 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:57.465631 (+ 290us) client_negotiation.cc:769] Sending connection context
1027 14:51:57.465915 (+ 284us) client_negotiation.cc:240] Negotiation successful
1027 14:51:57.466387 (+ 472us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":489,"thread_start_us":323,"threads_started":1}
I20251027 14:51:57.468992 7687 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44599
I20251027 14:51:57.469363 7687 heartbeater.cc:461] Registering TS with master...
I20251027 14:51:57.470299 7687 heartbeater.cc:507] Master 127.0.104.62:44599 requested a full tablet report, sending...
I20251027 14:51:57.471782 7225 ts_manager.cc:194] Registered new tserver with Master: af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563)
I20251027 14:51:57.472577 7225 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:57599
I20251027 14:51:57.482002 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:51:57.563572 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:37569
--webserver_interface=127.0.104.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44599,127.0.104.61:37569
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:51:57.885453 7695 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:57.885767 7695 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:57.885861 7695 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:57.893885 7695 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:57.893991 7695 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:57.894047 7695 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:57.894093 7695 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:57.894147 7695 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:57.904433 7695 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44599,127.0.104.61:37569
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:37569
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
W20251027 14:51:57.904738 7695 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.104.62:44599,127.0.104.61:37569'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20251027 14:51:57.906473 7695 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:57.908149 7695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:57.916738 7701 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:57.916991 7700 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:57.917002 7703 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:57.918367 7695 server_base.cc:1047] running on GCE node
I20251027 14:51:57.919101 7695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:57.920493 7695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:57.921695 7695 hybrid_clock.cc:648] HybridClock initialized: now 1761576717921641 us; error 46 us; skew 500 ppm
I20251027 14:51:57.922013 7695 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:57 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:57.925136 7695 init.cc:377] Logged in from keytab as oryx/127.0.104.61@KRBTEST.COM (short username oryx)
I20251027 14:51:57.926710 7695 webserver.cc:492] Webserver started at http://127.0.104.61:44175/ using document root <none> and password file <none>
I20251027 14:51:57.927544 7695 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:57.927718 7695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:57.928126 7695 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:57.930792 7695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/instance:
uuid: "1681ae8013c442a9aaef00ecede5f9ef"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "d8b8f17f0a3260e29b6388360e778710"
server_key_iv: "c9b9df012c7107909bf313df14bb2ad5"
server_key_version: "encryptionkey@0"
I20251027 14:51:57.931592 7695 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal/instance:
uuid: "1681ae8013c442a9aaef00ecede5f9ef"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "d8b8f17f0a3260e29b6388360e778710"
server_key_iv: "c9b9df012c7107909bf313df14bb2ad5"
server_key_version: "encryptionkey@0"
I20251027 14:51:57.934484 7552 heartbeater.cc:499] Master 127.0.104.62:44599 was elected leader, sending a full tablet report...
I20251027 14:51:57.937556 7695 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.002s
I20251027 14:51:57.940826 7710 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:57.942167 7695 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:51:57.942314 7695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
uuid: "1681ae8013c442a9aaef00ecede5f9ef"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "d8b8f17f0a3260e29b6388360e778710"
server_key_iv: "c9b9df012c7107909bf313df14bb2ad5"
server_key_version: "encryptionkey@0"
I20251027 14:51:57.942483 7695 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:57.968469 7695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:57.972399 7695 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:57.972644 7695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:57.990610 7695 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:37569
I20251027 14:51:57.990628 7761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:37569 every 8 connection(s)
I20251027 14:51:57.992028 7695 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
I20251027 14:51:57.992460 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7695
I20251027 14:51:57.996876 7762 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:58.003177 7762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
I20251027 14:51:58.013235 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:57.994700 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40632 (local address 127.0.104.61:37569)
1027 14:51:57.995471 (+ 771us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:57.995483 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:57.995527 (+ 44us) server_negotiation.cc:408] Connection header received
1027 14:51:57.996836 (+ 1309us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:57.996859 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:57.997316 (+ 457us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:57.997693 (+ 377us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:57.999310 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.000725 (+ 1415us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.001824 (+ 1099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.002343 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.005807 (+ 3464us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.005931 (+ 124us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.005951 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.005996 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.009516 (+ 3520us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.010111 (+ 595us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.010120 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.010128 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.010249 (+ 121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.010484 (+ 235us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.010489 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.010493 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.011034 (+ 541us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.011279 (+ 245us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.011690 (+ 411us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.012168 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":394,"threads_started":1}
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.61@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:51:58.018843 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.005442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47540 (local address 127.0.104.62:44599)
1027 14:51:58.005939 (+ 497us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.005946 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.006422 (+ 476us) server_negotiation.cc:408] Connection header received
1027 14:51:58.006602 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.006608 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.006687 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.006809 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.008530 (+ 1721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.009432 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.010664 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.010998 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.014127 (+ 3129us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.014160 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.014164 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.014191 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.016914 (+ 2723us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.017450 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.017454 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.017456 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.017537 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.017782 (+ 245us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.017785 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.017787 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.018017 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.018118 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.018445 (+ 327us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.018618 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":359,"thread_start_us":250,"threads_started":1}
I20251027 14:51:58.018868 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.005383 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47540)
1027 14:51:58.006280 (+ 897us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.006324 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.006472 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.006842 (+ 370us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.006847 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.007778 (+ 931us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.008358 (+ 580us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.008371 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.009571 (+ 1200us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.009576 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.010550 (+ 974us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.010561 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.010739 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.011833 (+ 1094us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.011850 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.013982 (+ 2132us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.017026 (+ 3044us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.017033 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.017038 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.017369 (+ 331us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.017608 (+ 239us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.017611 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.017613 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.017713 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.018102 (+ 389us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.018107 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.018229 (+ 122us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.018390 (+ 161us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.018589 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":674,"thread_start_us":546,"threads_started":1}
I20251027 14:51:58.020288 7762 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576717, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.61@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
I20251027 14:51:58.030185 7766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.020810 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:40648)
1027 14:51:58.020938 (+ 128us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.020954 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.021041 (+ 87us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.021279 (+ 238us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.021284 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.021560 (+ 276us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.021853 (+ 293us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.021862 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.022850 (+ 988us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.022854 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.023508 (+ 654us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.023518 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.023636 (+ 118us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.024168 (+ 532us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.024183 (+ 15us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.025825 (+ 1642us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.028335 (+ 2510us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.028340 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.028345 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.028660 (+ 315us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.028927 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.028930 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.028933 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.028993 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.029368 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.029372 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.029501 (+ 129us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.029608 (+ 107us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.029846 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":38}
I20251027 14:51:58.030310 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.020877 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40648 (local address 127.0.104.61:37569)
1027 14:51:58.020990 (+ 113us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.020995 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.021010 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:51:58.021128 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.021133 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.021182 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.021251 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.021973 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.022734 (+ 761us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.023620 (+ 886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.024020 (+ 400us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.025945 (+ 1925us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.025967 (+ 22us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.025970 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.025993 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.028225 (+ 2232us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.028754 (+ 529us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.028759 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.028762 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.028832 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.029073 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.029078 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.029081 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.029272 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.029386 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.029709 (+ 323us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.029988 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":66,"server-negotiator.queue_time_us":40}
I20251027 14:51:58.034786 7762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Bootstrap starting.
I20251027 14:51:58.037698 7762 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:58.038715 7762 log.cc:826] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:58.041199 7762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: No bootstrap required, opened a new log
I20251027 14:51:58.046041 7762 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } }
I20251027 14:51:58.046334 7762 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:58.046413 7762 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1681ae8013c442a9aaef00ecede5f9ef, State: Initialized, Role: FOLLOWER
I20251027 14:51:58.047019 7762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } }
I20251027 14:51:58.047819 7769 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } } }
I20251027 14:51:58.048033 7769 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:51:58.048794 7762 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: configured and running, proceeding with master startup.
W20251027 14:51:58.057091 7780 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 14:51:58.057333 7780 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 14:51:58.059551 7695 master_runner.cc:428] Detected that this master 1681ae8013c442a9aaef00ecede5f9ef is joining an existing cluster
I20251027 14:51:58.059664 7695 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.104.61:37569
I20251027 14:51:58.078501 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.064644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40650 (local address 127.0.104.61:37569)
1027 14:51:58.064800 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.064805 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.065360 (+ 555us) server_negotiation.cc:408] Connection header received
1027 14:51:58.065509 (+ 149us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.065518 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.065569 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.065645 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.066885 (+ 1240us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.068124 (+ 1239us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.069642 (+ 1518us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.070096 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.072107 (+ 2011us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.072138 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.072142 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.072167 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.075872 (+ 3705us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.076712 (+ 840us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.076717 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.076720 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.076782 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.077051 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.077055 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.077058 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.077349 (+ 291us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.077486 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.077893 (+ 407us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.078144 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:51:58.078859 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.063653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47550 (local address 127.0.104.62:44599)
1027 14:51:58.063830 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.063836 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.064443 (+ 607us) server_negotiation.cc:408] Connection header received
1027 14:51:58.064643 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.064649 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.064734 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.064821 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.066557 (+ 1736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.067884 (+ 1327us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.069663 (+ 1779us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.070148 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.072294 (+ 2146us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.072412 (+ 118us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.072420 (+ 8us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.072457 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.075815 (+ 3358us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.076806 (+ 991us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.076812 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.076816 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.076899 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.077297 (+ 398us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.077303 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.077306 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.077577 (+ 271us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.077719 (+ 142us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.078272 (+ 553us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.078513 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:51:58.079339 7787 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.064799 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:40650)
1027 14:51:58.065266 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.065282 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.065396 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.065667 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.065671 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.066228 (+ 557us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.066749 (+ 521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.066762 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.068284 (+ 1522us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.068289 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.069502 (+ 1213us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.069515 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.070285 (+ 770us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.070910 (+ 625us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.070939 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.071976 (+ 1037us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.076010 (+ 4034us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.076014 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.076018 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.076590 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.076880 (+ 290us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.076884 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.076887 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.076969 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.077566 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.077570 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.077687 (+ 117us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.078815 (+ 1128us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.079054 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":373,"thread_start_us":257,"threads_started":1}
I20251027 14:51:58.080013 7786 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.063679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47550)
1027 14:51:58.064328 (+ 649us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.064346 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.064494 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.065000 (+ 506us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.065009 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.065684 (+ 675us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.066366 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.066380 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.068110 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.068116 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.069499 (+ 1383us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.069514 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.070365 (+ 851us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.071073 (+ 708us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.071096 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.072129 (+ 1033us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.075972 (+ 3843us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.075977 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.075981 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.076659 (+ 678us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.077045 (+ 386us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.077050 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.077054 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.077179 (+ 125us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.077794 (+ 615us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.077798 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.077941 (+ 143us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.079290 (+ 1349us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.079553 (+ 263us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":520,"spinlock_wait_cycles":62208,"thread_start_us":385,"threads_started":1}
WARNING: no policy specified for oryx/127.0.104.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.60@KRBTEST.COM" created.
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.62@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
I20251027 14:51:58.103523 7789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.086627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:40664)
1027 14:51:58.087282 (+ 655us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.087319 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.087554 (+ 235us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.088014 (+ 460us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.088023 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.088900 (+ 877us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.089451 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.089465 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.091104 (+ 1639us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.091111 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.092211 (+ 1100us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.092231 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.092416 (+ 185us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.093142 (+ 726us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.093167 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.095509 (+ 2342us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.098599 (+ 3090us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.098612 (+ 13us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.098619 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.098961 (+ 342us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.099231 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.099235 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.099237 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.099307 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.102358 (+ 3051us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.102375 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.102574 (+ 199us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.102844 (+ 270us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.103173 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":303,"threads_started":1}
I20251027 14:51:58.103587 7765 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.086647 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40664 (local address 127.0.104.61:37569)
1027 14:51:58.086814 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.086820 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.087405 (+ 585us) server_negotiation.cc:408] Connection header received
1027 14:51:58.087753 (+ 348us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.087759 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.087843 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.087953 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.089614 (+ 1661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.090945 (+ 1331us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.092354 (+ 1409us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.092838 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.095654 (+ 2816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.095690 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.095694 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.095722 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.098458 (+ 2736us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.099068 (+ 610us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.099073 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.099075 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.099137 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.099409 (+ 272us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.099414 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.099418 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.099620 (+ 202us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.099764 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.102965 (+ 3201us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.103206 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:51:58.105760 7225 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.0.104.61:37569: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef"
member_type: NON_VOTER
last_known_addr {
host: "127.0.104.61"
port: 37569
}
attrs {
promote: true
}
}
dest_uuid: "ebe7c276923347a3ae5401422b9dab66"
cas_config_opid_index: -1
I20251027 14:51:58.106880 7769 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:51:58.106971 7225 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } }
I20251027 14:51:58.107101 7769 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } }
I20251027 14:51:58.108481 7769 leader_election.cc:290] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599)
I20251027 14:51:58.109416 7235 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1681ae8013c442a9aaef00ecede5f9ef" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe7c276923347a3ae5401422b9dab66" is_pre_election: true
I20251027 14:51:58.110524 7712 leader_election.cc:304] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 2 responses out of 2 voters: 1 yes votes; 1 no votes. yes voters: 1681ae8013c442a9aaef00ecede5f9ef; no voters: ebe7c276923347a3ae5401422b9dab66
I20251027 14:51:58.111227 7769 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 14:51:58.115356 7792 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61) added. New config: { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } }
I20251027 14:51:58.116186 7737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:58.119694 7792 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.0.104.61:37569
I20251027 14:51:58.120172 7792 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } }
I20251027 14:51:58.120466 7792 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:58.120559 7695 master.cc:561] Master@127.0.104.61:37569 shutting down...
I20251027 14:51:58.120752 7737 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Refusing update from remote peer ebe7c276923347a3ae5401422b9dab66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20251027 14:51:58.124526 7793 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 0, Time since last communication: 0.000s
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
W20251027 14:51:58.127427 7210 proxy.cc:239] Call had error, refreshing address and retrying: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master
W20251027 14:51:58.129009 7210 consensus_peers.cc:597] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 -> Peer 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61:37569): Couldn't send request to peer 1681ae8013c442a9aaef00ecede5f9ef. Status: Remote error: Service unavailable: service kudu.consensus.ConsensusService not registered on Master. This is attempt 1: this message will repeat every 5th retry.
WARNING: no policy specified for HTTP/127.0.104.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.60@KRBTEST.COM" created.
I20251027 14:51:58.160598 7695 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:51:58.161003 7695 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:58.161196 7695 tablet_replica.cc:333] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: stopping tablet replica
I20251027 14:51:58.168185 7695 master.cc:583] Master@127.0.104.61:37569 shutdown complete.
I20251027 14:51:58.168254 7695 master_runner.cc:305] Clearing existing system tablet
I20251027 14:51:58.168622 7695 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:58.168723 7695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:58.170810 7695 fs_manager.cc:714] Time spent opening directory manager: real 0.001s user 0.002s sys 0.000s
I20251027 14:51:58.172123 7799 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:58.172585 7695 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:58.172884 7695 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251027 14:51:58.173089 7695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
uuid: "1681ae8013c442a9aaef00ecede5f9ef"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "d8b8f17f0a3260e29b6388360e778710"
server_key_iv: "c9b9df012c7107909bf313df14bb2ad5"
server_key_version: "encryptionkey@0"
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:51:58.175850 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43831
--webserver_interface=127.0.104.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:51:58.177851 7695 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:58.185292 7695 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251027 14:51:58.185472 7695 log.cc:1199] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal/wals/00000000000000000000000000000000
I20251027 14:51:58.185986 7695 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Deleting consensus metadata
I20251027 14:51:58.186224 7695 master_runner.cc:315] Copying system tablet from 127.0.104.62:44599
I20251027 14:51:58.189216 7695 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.62:44599
I20251027 14:51:58.205442 7806 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.190065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47558)
1027 14:51:58.192793 (+ 2728us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.192813 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.192994 (+ 181us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.193437 (+ 443us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.193442 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.193995 (+ 553us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.194523 (+ 528us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.194536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.196212 (+ 1676us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.196219 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.197507 (+ 1288us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.197522 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.197669 (+ 147us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.198319 (+ 650us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.198342 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.199286 (+ 944us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.203021 (+ 3735us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.203027 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.203031 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.203467 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.203770 (+ 303us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.203775 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.203779 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.203875 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.204452 (+ 577us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.204458 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.204603 (+ 145us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.204765 (+ 162us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.205059 (+ 294us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2589,"thread_start_us":2476,"threads_started":1}
I20251027 14:51:58.205471 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.190066 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47558 (local address 127.0.104.62:44599)
1027 14:51:58.190239 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.190245 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.192968 (+ 2723us) server_negotiation.cc:408] Connection header received
1027 14:51:58.193163 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.193170 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.193274 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.193380 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.194695 (+ 1315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.196064 (+ 1369us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.197647 (+ 1583us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.198074 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.199459 (+ 1385us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.199511 (+ 52us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.199517 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.199552 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.202866 (+ 3314us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.203571 (+ 705us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.203577 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.203580 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.203662 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.203993 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.203998 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.204001 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.204280 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.204450 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.204891 (+ 441us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.205134 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:51:58.206470 7245 tablet_copy_service.cc:140] P ebe7c276923347a3ae5401422b9dab66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 1681ae8013c442a9aaef00ecede5f9ef ({username='oryx', principal='oryx/127.0.104.61@KRBTEST.COM'} at 127.0.0.1:47558)
I20251027 14:51:58.206735 7245 tablet_copy_service.cc:161] P ebe7c276923347a3ae5401422b9dab66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 1681ae8013c442a9aaef00ecede5f9ef at {username='oryx', principal='oryx/127.0.104.61@KRBTEST.COM'} at 127.0.0.1:47558: session id = 1681ae8013c442a9aaef00ecede5f9ef-00000000000000000000000000000000
I20251027 14:51:58.210903 7245 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 14:51:58.213191 7695 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251027 14:51:58.220180 7695 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: tablet copy: Starting download of 0 data blocks...
I20251027 14:51:58.220494 7695 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: tablet copy: Starting download of 1 WAL segments...
I20251027 14:51:58.223455 7695 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 14:51:58.227919 7245 tablet_copy_service.cc:342] P ebe7c276923347a3ae5401422b9dab66: Request end of tablet copy session 1681ae8013c442a9aaef00ecede5f9ef-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.104.61@KRBTEST.COM'} at 127.0.0.1:47558
I20251027 14:51:58.228138 7245 tablet_copy_service.cc:434] P ebe7c276923347a3ae5401422b9dab66: ending tablet copy session 1681ae8013c442a9aaef00ecede5f9ef-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 1681ae8013c442a9aaef00ecede5f9ef
W20251027 14:51:58.236987 7695 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251027 14:51:58.240913 7695 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:58.243464 7807 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:58.253293 7808 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:58.253909 7810 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:58.254554 7695 server_base.cc:1047] running on GCE node
I20251027 14:51:58.254900 7695 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:58.255486 7695 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:58.256762 7695 hybrid_clock.cc:648] HybridClock initialized: now 1761576718256690 us; error 46 us; skew 500 ppm
I20251027 14:51:58.257095 7695 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:58.259507 7695 init.cc:377] Logged in from keytab as oryx/127.0.104.61@KRBTEST.COM (short username oryx)
I20251027 14:51:58.260826 7695 webserver.cc:492] Webserver started at http://127.0.104.61:40087/ using document root <none> and password file <none>
I20251027 14:51:58.261271 7695 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:58.261351 7695 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:58.264263 7695 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251027 14:51:58.266050 7817 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:58.266474 7695 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.000s sys 0.001s
I20251027 14:51:58.266593 7695 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
uuid: "1681ae8013c442a9aaef00ecede5f9ef"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "d8b8f17f0a3260e29b6388360e778710"
server_key_iv: "c9b9df012c7107909bf313df14bb2ad5"
server_key_version: "encryptionkey@0"
I20251027 14:51:58.266749 7695 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:58.293473 7695 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:58.297405 7695 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:58.318763 7695 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:37569
I20251027 14:51:58.318936 7868 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:37569 every 8 connection(s)
I20251027 14:51:58.319864 7695 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
I20251027 14:51:58.324668 7869 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:51:58.328146 7869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Bootstrap starting.
I20251027 14:51:58.344725 7869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:51:58.345243 7869 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: Bootstrap complete.
I20251027 14:51:58.346351 7869 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } }
I20251027 14:51:58.346576 7869 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 1681ae8013c442a9aaef00ecede5f9ef, State: Initialized, Role: LEARNER
I20251027 14:51:58.346899 7869 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } }
I20251027 14:51:58.347661 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } }
I20251027 14:51:58.347975 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: LEARNER
I20251027 14:51:58.348037 7869 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:58.356201 7883 catalog_manager.cc:1269] Loaded cluster ID: 1d1d091730ea451bb60f617614e64065
I20251027 14:51:58.356300 7883 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: loading cluster ID for follower catalog manager: success
I20251027 14:51:58.360062 7883 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: acquiring CA information for follower catalog manager: success
I20251027 14:51:58.362506 7883 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 14:51:58.475014 7687 heartbeater.cc:499] Master 127.0.104.62:44599 was elected leader, sending a full tablet report...
W20251027 14:51:58.537245 7800 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:51:58.537561 7800 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:51:58.537691 7800 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:51:58.546031 7800 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:51:58.546140 7800 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:51:58.546222 7800 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:51:58.546283 7800 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:51:58.546362 7800 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:51:58.557554 7800 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43831
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:51:58.559088 7800 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:51:58.560415 7800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:58.568038 7889 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:58.568522 7888 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:58.568811 7891 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:58.569449 7800 server_base.cc:1047] running on GCE node
I20251027 14:51:58.570245 7800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:58.571609 7800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:58.572822 7800 hybrid_clock.cc:648] HybridClock initialized: now 1761576718572761 us; error 41 us; skew 500 ppm
I20251027 14:51:58.573155 7800 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:58.576288 7800 init.cc:377] Logged in from keytab as oryx/127.0.104.60@KRBTEST.COM (short username oryx)
I20251027 14:51:58.577816 7800 webserver.cc:492] Webserver started at http://127.0.104.60:46075/ using document root <none> and password file <none>
I20251027 14:51:58.578480 7800 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:58.578554 7800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:58.578850 7800 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:51:58.581368 7800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/instance:
uuid: "d93ee0148b8e47c09454292e9e2db849"
format_stamp: "Formatted at 2025-10-27 14:51:58 on dist-test-slave-hxxt"
server_key: "e1702f3eb3a2ca7ad04778510b22f512"
server_key_iv: "3f82efc4c5fda7d2fa27d74997fc95f6"
server_key_version: "encryptionkey@0"
I20251027 14:51:58.582095 7800 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal/instance:
uuid: "d93ee0148b8e47c09454292e9e2db849"
format_stamp: "Formatted at 2025-10-27 14:51:58 on dist-test-slave-hxxt"
server_key: "e1702f3eb3a2ca7ad04778510b22f512"
server_key_iv: "3f82efc4c5fda7d2fa27d74997fc95f6"
server_key_version: "encryptionkey@0"
I20251027 14:51:58.587198 7800 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:51:58.590420 7898 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:58.591742 7800 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:51:58.591892 7800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
uuid: "d93ee0148b8e47c09454292e9e2db849"
format_stamp: "Formatted at 2025-10-27 14:51:58 on dist-test-slave-hxxt"
server_key: "e1702f3eb3a2ca7ad04778510b22f512"
server_key_iv: "3f82efc4c5fda7d2fa27d74997fc95f6"
server_key_version: "encryptionkey@0"
I20251027 14:51:58.592095 7800 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:58.632095 7800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:58.636152 7800 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:51:58.636412 7800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:58.655287 7800 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:43831
I20251027 14:51:58.655294 7949 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:43831 every 8 connection(s)
I20251027 14:51:58.656709 7800 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
I20251027 14:51:58.657482 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 7800
I20251027 14:51:58.662115 7950 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:51:58.669016 7950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.60@KRBTEST.COM
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.60@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:51:58.678805 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.659723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34024 (local address 127.0.104.60:43831)
1027 14:51:58.660471 (+ 748us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.660483 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.660527 (+ 44us) server_negotiation.cc:408] Connection header received
1027 14:51:58.661890 (+ 1363us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.661912 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.662319 (+ 407us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.662737 (+ 418us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.664278 (+ 1541us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.665669 (+ 1391us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.666792 (+ 1123us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.667881 (+ 1089us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.670837 (+ 2956us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.670951 (+ 114us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.670971 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.671022 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.674691 (+ 3669us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.675283 (+ 592us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.675290 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.675296 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.675400 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.675736 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.675739 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.675741 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.676173 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.676424 (+ 251us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.677010 (+ 586us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.677593 (+ 583us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":516,"thread_start_us":338,"threads_started":1}
I20251027 14:51:58.684206 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.671218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47574 (local address 127.0.104.62:44599)
1027 14:51:58.671397 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.671403 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.672056 (+ 653us) server_negotiation.cc:408] Connection header received
1027 14:51:58.672252 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.672259 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.672337 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.672454 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.674026 (+ 1572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.675140 (+ 1114us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.676368 (+ 1228us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.677329 (+ 961us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.679437 (+ 2108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.679468 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.679471 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.679501 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.682030 (+ 2529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.682566 (+ 536us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.682571 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.682574 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.682635 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.682931 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.682936 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.682937 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.683163 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.683290 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.683698 (+ 408us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.683915 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:51:58.684252 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.671110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47574)
1027 14:51:58.671906 (+ 796us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.671947 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.672116 (+ 169us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.672524 (+ 408us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.672531 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.673310 (+ 779us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.673863 (+ 553us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.673890 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.675273 (+ 1383us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.675279 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.676233 (+ 954us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.676247 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.676434 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.677054 (+ 620us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.677078 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.679297 (+ 2219us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.682148 (+ 2851us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.682159 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.682165 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.682463 (+ 298us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.682730 (+ 267us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.682733 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.682735 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.682811 (+ 76us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.683272 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.683279 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.683429 (+ 150us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.683593 (+ 164us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.683885 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":561,"thread_start_us":428,"threads_started":1}
I20251027 14:51:58.685447 7950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.60@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
I20251027 14:51:58.696480 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.685943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:40674)
1027 14:51:58.686101 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.686119 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.686211 (+ 92us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.686857 (+ 646us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.686861 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.687110 (+ 249us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.687405 (+ 295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.687414 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.688840 (+ 1426us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.688844 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.689549 (+ 705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.689559 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.689684 (+ 125us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.690194 (+ 510us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.690210 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.691843 (+ 1633us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.694905 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.694909 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.694912 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.695181 (+ 269us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.695384 (+ 203us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.695387 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.695389 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.695443 (+ 54us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.695795 (+ 352us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.695799 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.695942 (+ 143us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.696044 (+ 102us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.696227 (+ 183us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":55}
I20251027 14:51:58.696545 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.686164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40674 (local address 127.0.104.61:37569)
1027 14:51:58.686591 (+ 427us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.686596 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.686610 (+ 14us) server_negotiation.cc:408] Connection header received
1027 14:51:58.686667 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.686671 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.686727 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.686814 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.687535 (+ 721us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.688735 (+ 1200us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.689673 (+ 938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.689998 (+ 325us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.691980 (+ 1982us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.692015 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.692018 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.692045 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.694806 (+ 2761us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.695263 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.695267 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.695269 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.695322 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.695502 (+ 180us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.695504 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.695506 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.695690 (+ 184us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.695799 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.696110 (+ 311us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.696271 (+ 161us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20251027 14:51:58.697553 7950 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.60@KRBTEST.COM for oryx/127.0.104.60@KRBTEST.COM
I20251027 14:51:58.707499 7954 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.698202 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34034)
1027 14:51:58.698325 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.698342 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.698440 (+ 98us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.698719 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.698723 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.698969 (+ 246us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.699256 (+ 287us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.699264 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.700242 (+ 978us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.700246 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.700914 (+ 668us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.700925 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.701038 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.701567 (+ 529us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.701584 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.703280 (+ 1696us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.705887 (+ 2607us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.705891 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.705893 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.706189 (+ 296us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.706403 (+ 214us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.706406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.706408 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.706464 (+ 56us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.706815 (+ 351us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.706818 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.706939 (+ 121us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.707036 (+ 97us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.707208 (+ 172us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251027 14:51:58.707549 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.698175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34034 (local address 127.0.104.60:43831)
1027 14:51:58.698327 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.698331 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.698395 (+ 64us) server_negotiation.cc:408] Connection header received
1027 14:51:58.698520 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.698525 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.698580 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.698689 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.699367 (+ 678us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.700134 (+ 767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.701027 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.701364 (+ 337us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.703414 (+ 2050us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.703439 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.703442 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.703466 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.705775 (+ 2309us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.706275 (+ 500us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.706279 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.706281 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.706334 (+ 53us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.706523 (+ 189us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.706526 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.706528 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.706725 (+ 197us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.706830 (+ 105us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.707115 (+ 285us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.707299 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":20,"server-negotiator.queue_time_us":50}
I20251027 14:51:58.712250 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Bootstrap starting.
I20251027 14:51:58.716533 7950 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Neither blocks nor log segments found. Creating new log.
I20251027 14:51:58.717532 7950 log.cc:826] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Log is configured to *not* fsync() on all Append() calls
I20251027 14:51:58.719780 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.715138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:40686)
1027 14:51:58.715585 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.715600 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.715738 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.716160 (+ 422us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.716165 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.716191 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:58.716653 (+ 462us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.716666 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.717765 (+ 1099us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.717772 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.719082 (+ 1310us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.719097 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.719249 (+ 152us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.719308 (+ 59us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.719372 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.719494 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":210,"threads_started":1}
I20251027 14:51:58.720522 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.715277 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40686 (local address 127.0.104.61:37569)
1027 14:51:58.715449 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.715456 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.715687 (+ 231us) server_negotiation.cc:408] Connection header received
1027 14:51:58.715887 (+ 200us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.715891 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.715982 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.716097 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:58.716785 (+ 688us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.717633 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.719221 (+ 1588us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.720026 (+ 805us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.720156 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.720233 (+ 77us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.720323 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:51:58.720655 7950 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: No bootstrap required, opened a new log
I20251027 14:51:58.721565 7844 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 LEARNER]: Deduplicated request from leader. Original: 0.0->[1.1-1.5] Dedup: 1.5->[]
I20251027 14:51:58.722114 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe7c276923347a3ae5401422b9dab66. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 5 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } }
I20251027 14:51:58.722294 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: LEARNER
I20251027 14:51:58.726449 7950 raft_consensus.cc:359] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } }
I20251027 14:51:58.726747 7950 raft_consensus.cc:385] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:51:58.726867 7950 raft_consensus.cc:740] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d93ee0148b8e47c09454292e9e2db849, State: Initialized, Role: FOLLOWER
I20251027 14:51:58.727414 7950 consensus_queue.cc:260] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } }
I20251027 14:51:58.728189 7959 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } } }
I20251027 14:51:58.728430 7959 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:51:58.729116 7950 sys_catalog.cc:565] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: configured and running, proceeding with master startup.
W20251027 14:51:58.737892 7970 catalog_manager.cc:1568] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 14:51:58.738022 7970 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 14:51:58.741591 7800 master_runner.cc:428] Detected that this master d93ee0148b8e47c09454292e9e2db849 is joining an existing cluster
I20251027 14:51:58.741674 7800 master_runner.cc:432] Initiating AddMaster RPC to add 127.0.104.60:43831
I20251027 14:51:58.760619 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.746014 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47578 (local address 127.0.104.62:44599)
1027 14:51:58.746186 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.746193 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.747691 (+ 1498us) server_negotiation.cc:408] Connection header received
1027 14:51:58.747900 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.747907 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.747979 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.748046 (+ 67us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.749661 (+ 1615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.751400 (+ 1739us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.753368 (+ 1968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.753832 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.754897 (+ 1065us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.754939 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.754945 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.754980 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.757623 (+ 2643us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.758380 (+ 757us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.758386 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.758390 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.758496 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.758932 (+ 436us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.758937 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.758940 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.759209 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.759358 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.759956 (+ 598us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.760213 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:51:58.761166 7977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.746494 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47578)
1027 14:51:58.747614 (+ 1120us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.747628 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.747755 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.748252 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.748256 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.748939 (+ 683us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.749497 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.749506 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.751626 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.751632 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.752918 (+ 1286us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.752931 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.753082 (+ 151us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.753748 (+ 666us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.753772 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.754743 (+ 971us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.757761 (+ 3018us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.757766 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.757771 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.758239 (+ 468us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.758679 (+ 440us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.758685 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.758689 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.758800 (+ 111us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.759412 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.759416 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.759587 (+ 171us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.759735 (+ 148us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.760709 (+ 974us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1017,"thread_start_us":913,"threads_started":1}
I20251027 14:51:58.766667 7955 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.747857 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40702 (local address 127.0.104.61:37569)
1027 14:51:58.748012 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.748017 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.754461 (+ 6444us) server_negotiation.cc:408] Connection header received
1027 14:51:58.754619 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.754626 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.754705 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.754805 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.756073 (+ 1268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.757417 (+ 1344us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.759517 (+ 2100us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.759998 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.761489 (+ 1491us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.761521 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.761525 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.761554 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.764324 (+ 2770us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.765079 (+ 755us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.765083 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.765085 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.765150 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.765444 (+ 294us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.765447 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.765449 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.765640 (+ 191us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.765803 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.766137 (+ 334us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.766341 (+ 204us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:51:58.766734 7978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.748129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:40702)
1027 14:51:58.754345 (+ 6216us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.754360 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.754476 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.754845 (+ 369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.754850 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.755393 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.755922 (+ 529us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.755935 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.757596 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.757602 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.759384 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.759396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.759558 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.760140 (+ 582us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.760162 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.761339 (+ 1177us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.764493 (+ 3154us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.764499 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.764503 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.764976 (+ 473us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.765246 (+ 270us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.765251 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.765255 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.765346 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.765762 (+ 416us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.765767 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.765905 (+ 138us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.766032 (+ 127us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.766284 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6106,"thread_start_us":5922,"threads_started":1}
I20251027 14:51:58.773877 7959 raft_consensus.cc:493] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:51:58.774132 7959 raft_consensus.cc:515] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } }
I20251027 14:51:58.775673 7959 leader_election.cc:290] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61:37569)
I20251027 14:51:58.776567 7235 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93ee0148b8e47c09454292e9e2db849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ebe7c276923347a3ae5401422b9dab66" is_pre_election: true
I20251027 14:51:58.777040 7844 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93ee0148b8e47c09454292e9e2db849" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1681ae8013c442a9aaef00ecede5f9ef" is_pre_election: true
I20251027 14:51:58.778119 7900 leader_election.cc:304] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d93ee0148b8e47c09454292e9e2db849; no voters: 1681ae8013c442a9aaef00ecede5f9ef, ebe7c276923347a3ae5401422b9dab66
I20251027 14:51:58.778810 7959 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20251027 14:51:58.780875 7976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.746236 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34048)
1027 14:51:58.746833 (+ 597us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.746855 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.747484 (+ 629us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.747933 (+ 449us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.747938 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.748673 (+ 735us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.749365 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.749381 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.751142 (+ 1761us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.751147 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.752324 (+ 1177us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.752344 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.753136 (+ 792us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.769517 (+ 16381us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.769539 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.770286 (+ 747us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.778252 (+ 7966us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.778258 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.778262 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.778754 (+ 492us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.779204 (+ 450us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.779209 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.779213 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.779294 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.779805 (+ 511us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.779810 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.779962 (+ 152us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.780103 (+ 141us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.780403 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":484,"thread_start_us":347,"threads_started":1}
I20251027 14:51:58.780963 7953 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.746112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34048 (local address 127.0.104.60:43831)
1027 14:51:58.746280 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.746286 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.747385 (+ 1099us) server_negotiation.cc:408] Connection header received
1027 14:51:58.747633 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.747639 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.747702 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.747759 (+ 57us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.749500 (+ 1741us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.751003 (+ 1503us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.752488 (+ 1485us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.752948 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.770436 (+ 17488us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.770478 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.770483 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.770512 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.774970 (+ 4458us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.778887 (+ 3917us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.778896 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.778899 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.779067 (+ 168us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.779405 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.779411 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.779415 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.779691 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.779817 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.780215 (+ 398us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.780548 (+ 333us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":100,"server-negotiator.queue_time_us":72}
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.62@KRBTEST.COM for oryx/127.0.104.60@KRBTEST.COM
I20251027 14:51:58.798259 7957 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.772911 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34058)
1027 14:51:58.773079 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.773094 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.773175 (+ 81us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.783614 (+ 10439us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.783620 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.784103 (+ 483us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.784539 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.784555 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.786635 (+ 2080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.786642 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.787869 (+ 1227us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.787884 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.787994 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.788764 (+ 770us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.788806 (+ 42us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.791532 (+ 2726us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.796067 (+ 4535us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.796072 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.796076 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.796484 (+ 408us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.796764 (+ 280us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.796773 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.796775 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.796846 (+ 71us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.797367 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.797371 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.797496 (+ 125us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.797639 (+ 143us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.797903 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":48}
I20251027 14:51:58.798389 7979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.774118 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34058 (local address 127.0.104.60:43831)
1027 14:51:58.778360 (+ 4242us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.778366 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.778386 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:51:58.778469 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.778474 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.778552 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.783373 (+ 4821us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.784740 (+ 1367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.786453 (+ 1713us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.791742 (+ 5289us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.792242 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.792416 (+ 174us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.792480 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.792485 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.792515 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.795944 (+ 3429us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.796588 (+ 644us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.796592 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.796594 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.796665 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.796961 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.796964 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.796966 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.797193 (+ 227us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.797320 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.797809 (+ 489us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.798033 (+ 224us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4124,"thread_start_us":3995,"threads_started":1}
I20251027 14:51:58.800408 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.786071 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47590)
1027 14:51:58.793317 (+ 7246us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.793334 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.793461 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.793887 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.793892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.793911 (+ 19us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:58.794537 (+ 626us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.794552 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.796220 (+ 1668us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.796226 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.798203 (+ 1977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.798219 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.799159 (+ 940us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.799187 (+ 28us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.799258 (+ 71us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.799376 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7073,"thread_start_us":6434,"threads_started":1}
I20251027 14:51:58.800957 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.786335 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47590 (local address 127.0.104.62:44599)
1027 14:51:58.786555 (+ 220us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.786560 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.793509 (+ 6949us) server_negotiation.cc:408] Connection header received
1027 14:51:58.793585 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.793591 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.793667 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.793745 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:58.794668 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.795996 (+ 1328us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.799090 (+ 3094us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.800554 (+ 1464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.800629 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.800690 (+ 61us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.800803 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20251027 14:51:58.801487 7225 catalog_manager.cc:7136] Initiating ChangeConfig request to add master 127.0.104.60:43831: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "d93ee0148b8e47c09454292e9e2db849"
member_type: NON_VOTER
last_known_addr {
host: "127.0.104.60"
port: 43831
}
attrs {
promote: true
}
}
dest_uuid: "ebe7c276923347a3ae5401422b9dab66"
cas_config_opid_index: 5
I20251027 14:51:58.802520 7225 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } }
I20251027 14:51:58.804193 7956 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d93ee0148b8e47c09454292e9e2db849 (127.0.104.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.805022 7925 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:51:58.808068 7956 catalog_manager.cc:7140] Successfully completed master ChangeConfig request to add master 127.0.104.60:43831
I20251027 14:51:58.808647 7925 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Refusing update from remote peer ebe7c276923347a3ae5401422b9dab66: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251027 14:51:58.808737 7800 master.cc:561] Master@127.0.104.60:43831 shutting down...
I20251027 14:51:58.809033 7844 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 LEARNER]: Refusing update from remote peer ebe7c276923347a3ae5401422b9dab66: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20251027 14:51:58.816465 7956 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:51:58.819849 7956 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20251027 14:51:58.820740 7800 raft_consensus.cc:2243] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:51:58.821146 7800 raft_consensus.cc:2272] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:51:58.821312 7800 tablet_replica.cc:333] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: stopping tablet replica
I20251027 14:51:58.824411 7981 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.824903 7981 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:58.831380 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.830968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34068)
1027 14:51:58.831108 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.831204 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251027 14:51:58.831692 7212 consensus_peers.cc:597] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 -> Peer d93ee0148b8e47c09454292e9e2db849 (127.0.104.60:43831): Couldn't send request to peer d93ee0148b8e47c09454292e9e2db849. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 14:51:58.834297 7800 master.cc:583] Master@127.0.104.60:43831 shutdown complete.
I20251027 14:51:58.834386 7800 master_runner.cc:305] Clearing existing system tablet
I20251027 14:51:58.833317 7844 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 LEARNER]: Committing config change with OpId 1.6: config changed from index 5 to 6, NON_VOTER d93ee0148b8e47c09454292e9e2db849 (127.0.104.60) added. New config: { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.834833 7800 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:58.834929 7800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:58.837886 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 6 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: NON_VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: true } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.838160 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: LEARNER
I20251027 14:51:58.839020 7983 raft_consensus.cc:1064] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: attempting to promote NON_VOTER 1681ae8013c442a9aaef00ecede5f9ef to VOTER
I20251027 14:51:58.839597 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.839196 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34070)
1027 14:51:58.839331 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.839408 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251027 14:51:58.839818 7800 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
W20251027 14:51:58.839984 7224 master.cc:679] Network error: unable to get registration information for peer d93ee0148b8e47c09454292e9e2db849: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
I20251027 14:51:58.840004 7983 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 1.6, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } }
I20251027 14:51:58.845053 7844 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 LEARNER]: Refusing update from remote peer ebe7c276923347a3ae5401422b9dab66: Log matching property violated. Preceding OpId in replica: term: 1 index: 6. Preceding OpId from leader: term: 1 index: 7. (index mismatch)
I20251027 14:51:58.845625 7981 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20251027 14:51:58.846138 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.845765 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34072)
1027 14:51:58.845906 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.845979 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20251027 14:51:58.847148 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.846739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34086)
1027 14:51:58.846921 (+ 182us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.846989 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":98}
W20251027 14:51:58.847321 7212 consensus_peers.cc:597] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 -> Peer d93ee0148b8e47c09454292e9e2db849 (127.0.104.60:43831): Couldn't send request to peer d93ee0148b8e47c09454292e9e2db849. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 14:51:58.847776 7956 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 LEADER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.848050 7988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:58.848425 7800 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:58.849421 7800 fs_manager.cc:730] Time spent opening block manager: real 0.009s user 0.005s sys 0.001s
I20251027 14:51:58.849514 7800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
uuid: "d93ee0148b8e47c09454292e9e2db849"
format_stamp: "Formatted at 2025-10-27 14:51:58 on dist-test-slave-hxxt"
server_key: "e1702f3eb3a2ca7ad04778510b22f512"
server_key_iv: "3f82efc4c5fda7d2fa27d74997fc95f6"
server_key_version: "encryptionkey@0"
I20251027 14:51:58.852001 7844 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Committing config change with OpId 1.7: config changed from index 6 to 7, 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.853961 7800 ts_tablet_manager.cc:1916] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:51:58.856712 7981 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.856927 7981 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:58.857062 7956 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.857211 7956 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:51:58.857618 7872 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.857810 7872 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:51:58.860405 7800 ts_tablet_manager.cc:1929] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20251027 14:51:58.860507 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.860082 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34102)
1027 14:51:58.860231 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.860314 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251027 14:51:58.860594 7800 log.cc:1199] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal/wals/00000000000000000000000000000000
W20251027 14:51:58.860932 7224 master.cc:679] Network error: unable to get registration information for peer d93ee0148b8e47c09454292e9e2db849: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
I20251027 14:51:58.861176 7800 ts_tablet_manager.cc:1950] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Deleting consensus metadata
I20251027 14:51:58.861451 7800 master_runner.cc:315] Copying system tablet from 127.0.104.62:44599
I20251027 14:51:58.863693 7800 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: tablet copy: Beginning tablet copy session from remote peer at address 127.0.104.62:44599
I20251027 14:51:58.871083 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.870618 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34116)
1027 14:51:58.870815 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.870896 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":91}
W20251027 14:51:58.871425 7224 master.cc:679] Network error: unable to get registration information for peer d93ee0148b8e47c09454292e9e2db849: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
I20251027 14:51:58.876333 7994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.864263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:47592)
1027 14:51:58.864778 (+ 515us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.864806 (+ 28us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:58.864950 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:58.865234 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:58.865240 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:58.865821 (+ 581us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:51:58.866366 (+ 545us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.866379 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.868295 (+ 1916us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.868300 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:58.869349 (+ 1049us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:58.869362 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.869543 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.870124 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:51:58.870143 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:51:58.870951 (+ 808us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:51:58.874453 (+ 3502us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.874457 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.874460 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.874772 (+ 312us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.875050 (+ 278us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:51:58.875053 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:51:58.875055 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:51:58.875119 (+ 64us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:51:58.875622 (+ 503us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:51:58.875625 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:51:58.875733 (+ 108us) client_negotiation.cc:769] Sending connection context
1027 14:51:58.875842 (+ 109us) client_negotiation.cc:240] Negotiation successful
1027 14:51:58.876052 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":278,"threads_started":1}
I20251027 14:51:58.876520 7767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.864291 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47592 (local address 127.0.104.62:44599)
1027 14:51:58.864469 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:58.864476 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:58.864899 (+ 423us) server_negotiation.cc:408] Connection header received
1027 14:51:58.865061 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:58.865065 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:58.865125 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:58.865194 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:51:58.866522 (+ 1328us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.868139 (+ 1617us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:58.869527 (+ 1388us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:58.870007 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:58.871082 (+ 1075us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:51:58.871118 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:51:58.871122 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:51:58.871149 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:51:58.874334 (+ 3185us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.874873 (+ 539us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.874880 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.874884 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.874962 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:51:58.875208 (+ 246us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:51:58.875213 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:51:58.875217 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:51:58.875493 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:51:58.875633 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:58.875957 (+ 324us) server_negotiation.cc:300] Negotiation successful
1027 14:51:58.876184 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:51:58.876905 7245 tablet_copy_service.cc:140] P ebe7c276923347a3ae5401422b9dab66: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer d93ee0148b8e47c09454292e9e2db849 ({username='oryx', principal='oryx/127.0.104.60@KRBTEST.COM'} at 127.0.0.1:47592)
I20251027 14:51:58.877046 7245 tablet_copy_service.cc:161] P ebe7c276923347a3ae5401422b9dab66: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer d93ee0148b8e47c09454292e9e2db849 at {username='oryx', principal='oryx/127.0.104.60@KRBTEST.COM'} at 127.0.0.1:47592: session id = d93ee0148b8e47c09454292e9e2db849-00000000000000000000000000000000
I20251027 14:51:58.879011 7245 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Tablet Copy: opened 0 blocks and 1 log segments
I20251027 14:51:58.880563 7800 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20251027 14:51:58.884286 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.883879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34130)
1027 14:51:58.884076 (+ 197us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.884153 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":104}
W20251027 14:51:58.884572 7224 master.cc:679] Network error: unable to get registration information for peer d93ee0148b8e47c09454292e9e2db849: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
I20251027 14:51:58.886489 7800 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: tablet copy: Starting download of 0 data blocks...
I20251027 14:51:58.886745 7800 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: tablet copy: Starting download of 1 WAL segments...
I20251027 14:51:58.888751 7800 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20251027 14:51:58.891732 7245 tablet_copy_service.cc:342] P ebe7c276923347a3ae5401422b9dab66: Request end of tablet copy session d93ee0148b8e47c09454292e9e2db849-00000000000000000000000000000000 received from {username='oryx', principal='oryx/127.0.104.60@KRBTEST.COM'} at 127.0.0.1:47592
I20251027 14:51:58.891834 7245 tablet_copy_service.cc:434] P ebe7c276923347a3ae5401422b9dab66: ending tablet copy session d93ee0148b8e47c09454292e9e2db849-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer d93ee0148b8e47c09454292e9e2db849
W20251027 14:51:58.898074 7800 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20251027 14:51:58.901199 7800 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:51:58.903538 7995 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:51:58.903968 7996 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:58.905876 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.905430 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34136)
1027 14:51:58.905633 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.905709 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":102}
W20251027 14:51:58.906157 7224 master.cc:679] Network error: unable to get registration information for peer d93ee0148b8e47c09454292e9e2db849: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
I20251027 14:51:58.908646 7800 server_base.cc:1047] running on GCE node
W20251027 14:51:58.909417 7998 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:51:58.909803 7800 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:51:58.910415 7800 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:51:58.911599 7800 hybrid_clock.cc:648] HybridClock initialized: now 1761576718911564 us; error 28 us; skew 500 ppm
I20251027 14:51:58.911855 7800 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:51:58 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576718, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:51:58.914055 7800 init.cc:377] Logged in from keytab as oryx/127.0.104.60@KRBTEST.COM (short username oryx)
I20251027 14:51:58.915092 7800 webserver.cc:492] Webserver started at http://127.0.104.60:42017/ using document root <none> and password file <none>
I20251027 14:51:58.915418 7800 fs_manager.cc:362] Metadata directory not provided
I20251027 14:51:58.915513 7800 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:51:58.918213 7800 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20251027 14:51:58.919875 8005 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:51:58.920730 7800 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20251027 14:51:58.920845 7800 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
uuid: "d93ee0148b8e47c09454292e9e2db849"
format_stamp: "Formatted at 2025-10-27 14:51:58 on dist-test-slave-hxxt"
server_key: "e1702f3eb3a2ca7ad04778510b22f512"
server_key_iv: "3f82efc4c5fda7d2fa27d74997fc95f6"
server_key_version: "encryptionkey@0"
I20251027 14:51:58.921012 7800 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:51:58.937582 7800 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:51:58.941201 7800 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:51:58.943462 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:58.943003 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34138)
1027 14:51:58.943171 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:58.943286 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
W20251027 14:51:58.943763 7224 master.cc:679] Network error: unable to get registration information for peer d93ee0148b8e47c09454292e9e2db849: Client connection negotiation failed: client connection to 127.0.104.60:43831: connect: Connection refused (error 111)
I20251027 14:51:58.959704 7800 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:43831
I20251027 14:51:58.959720 8056 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:43831 every 8 connection(s)
I20251027 14:51:58.960563 7800 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
I20251027 14:51:58.965695 8057 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:51:58.969122 8057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Bootstrap starting.
I20251027 14:51:58.987844 8057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Bootstrap replayed 1/1 log segments. Stats: ops{read=7 overwritten=0 applied=7 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:51:58.988310 8057 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Bootstrap complete.
I20251027 14:51:58.989444 8057 raft_consensus.cc:359] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } }
I20251027 14:51:58.989678 8057 raft_consensus.cc:740] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: d93ee0148b8e47c09454292e9e2db849, State: Initialized, Role: LEARNER
I20251027 14:51:58.989926 8057 consensus_queue.cc:260] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 7, Last appended: 1.7, Last appended by leader: 7, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } }
I20251027 14:51:58.990640 8060 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:58.990854 8060 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: This master's current role is: LEARNER
I20251027 14:51:58.990924 8057 sys_catalog.cc:565] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:51:58.999023 8071 catalog_manager.cc:1269] Loaded cluster ID: 1d1d091730ea451bb60f617614e64065
I20251027 14:51:58.999114 8071 catalog_manager.cc:1562] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: loading cluster ID for follower catalog manager: success
I20251027 14:51:59.001579 8071 catalog_manager.cc:1584] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: acquiring CA information for follower catalog manager: success
I20251027 14:51:59.003325 8071 catalog_manager.cc:1612] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 14:51:59.017629 7980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:59.012358 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:34154)
1027 14:51:59.012560 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1027 14:51:59.012580 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:51:59.012715 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:51:59.013645 (+ 930us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:51:59.013649 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:51:59.013670 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:51:59.013987 (+ 317us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:59.013995 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:59.015987 (+ 1992us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:51:59.015992 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:51:59.017136 (+ 1144us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:51:59.017148 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:51:59.017254 (+ 106us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:59.017273 (+ 19us) client_negotiation.cc:769] Sending connection context
1027 14:51:59.017314 (+ 41us) client_negotiation.cc:240] Negotiation successful
1027 14:51:59.017406 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":76}
I20251027 14:51:59.019160 8072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:51:59.012537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34154 (local address 127.0.104.60:43831)
1027 14:51:59.013284 (+ 747us) server_negotiation.cc:207] Beginning negotiation
1027 14:51:59.013292 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:51:59.013314 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:51:59.013379 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:51:59.013386 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:51:59.013469 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:51:59.013618 (+ 149us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:51:59.014118 (+ 500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:59.015863 (+ 1745us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:51:59.017289 (+ 1426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:51:59.018449 (+ 1160us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:51:59.018630 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
1027 14:51:59.018728 (+ 98us) server_negotiation.cc:300] Negotiation successful
1027 14:51:59.018867 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":519,"threads_started":1}
I20251027 14:51:59.412930 8032 raft_consensus.cc:1217] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.6->[1.7-1.7] Dedup: 1.7->[]
I20251027 14:51:59.413511 8060 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ebe7c276923347a3ae5401422b9dab66. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:51:59.413733 8060 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: This master's current role is: LEARNER
I20251027 14:52:00.010425 8073 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 7 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: NON_VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: true } } }
I20251027 14:52:00.010819 8073 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:00.011749 8073 raft_consensus.cc:1064] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: attempting to promote NON_VOTER d93ee0148b8e47c09454292e9e2db849 to VOTER
I20251027 14:52:00.012396 8073 consensus_queue.cc:237] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 7, Committed index: 7, Last appended: 1.7, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:00.014324 8032 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 LEARNER]: Refusing update from remote peer ebe7c276923347a3ae5401422b9dab66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251027 14:52:00.014381 7844 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Refusing update from remote peer ebe7c276923347a3ae5401422b9dab66: Log matching property violated. Preceding OpId in replica: term: 1 index: 7. Preceding OpId from leader: term: 1 index: 8. (index mismatch)
I20251027 14:52:00.014938 8075 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251027 14:52:00.015242 8073 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 8, Last known committed idx: 7, Time since last communication: 0.000s
I20251027 14:52:00.018572 8075 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [term 1 LEADER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d93ee0148b8e47c09454292e9e2db849 (127.0.104.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.019354 8032 raft_consensus.cc:2955] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d93ee0148b8e47c09454292e9e2db849 (127.0.104.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.019840 7844 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Committing config change with OpId 1.8: config changed from index 7 to 8, d93ee0148b8e47c09454292e9e2db849 (127.0.104.60) changed from NON_VOTER to VOTER. New config: { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.022419 8075 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.022609 8075 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:00.022686 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.022899 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:00.023176 8074 sys_catalog.cc:455] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.023340 8074 sys_catalog.cc:458] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:00.023800 8080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.024389 8081 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ebe7c276923347a3ae5401422b9dab66" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:00.024650 8081 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:52:00.024828 8080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:52:00.951758 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.939519 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51348 (local address 127.0.104.61:37569)
1027 14:52:00.939975 (+ 456us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.939980 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.940001 (+ 21us) server_negotiation.cc:408] Connection header received
1027 14:52:00.940134 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.940138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.940195 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.940282 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:00.941924 (+ 1642us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.943310 (+ 1386us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.944726 (+ 1416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.945116 (+ 390us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.946724 (+ 1608us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:00.946786 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:00.946790 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:00.946820 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:00.949609 (+ 2789us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:00.950157 (+ 548us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:00.950161 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:00.950163 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:00.950227 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:00.950434 (+ 207us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:00.950437 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:00.950439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:00.950654 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:00.950832 (+ 178us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.951160 (+ 328us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.951392 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":365,"thread_start_us":259,"threads_started":1}
I20251027 14:52:00.958377 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.953611 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50178)
1027 14:52:00.954090 (+ 479us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:00.954105 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:00.954198 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:00.954557 (+ 359us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:00.954560 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:00.954582 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:00.955060 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.955068 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.956339 (+ 1271us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.956344 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:00.957635 (+ 1291us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.957650 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.957850 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.957877 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:00.957949 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:52:00.958069 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":247,"threads_started":1}
I20251027 14:52:00.959291 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.953770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50178 (local address 127.0.104.62:44599)
1027 14:52:00.954299 (+ 529us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.954305 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.954319 (+ 14us) server_negotiation.cc:408] Connection header received
1027 14:52:00.954372 (+ 53us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.954376 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.954440 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.954528 (+ 88us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:00.955195 (+ 667us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.956174 (+ 979us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.957847 (+ 1673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.958896 (+ 1049us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.958940 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.959002 (+ 62us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.959092 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":447,"thread_start_us":357,"threads_started":1}
I20251027 14:52:00.965312 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.960981 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:51358)
1027 14:52:00.961142 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:00.961160 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:00.961259 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:00.961534 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:00.961540 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:00.961570 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:00.961944 (+ 374us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.961954 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.963026 (+ 1072us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.963031 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:00.964660 (+ 1629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.964674 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.964810 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.964834 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:00.964903 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:52:00.965033 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":44}
I20251027 14:52:00.966192 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.961177 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51358 (local address 127.0.104.61:37569)
1027 14:52:00.961295 (+ 118us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.961299 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.961314 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:52:00.961375 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.961378 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.961436 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.961502 (+ 66us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:00.962099 (+ 597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.962907 (+ 808us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.964821 (+ 1914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.965856 (+ 1035us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.965894 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.965960 (+ 66us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.966043 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":38}
I20251027 14:52:00.971918 8084 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.967898 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:46720)
1027 14:52:00.968054 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:00.968072 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:00.968151 (+ 79us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:00.968772 (+ 621us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:00.968777 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:00.968804 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:00.969118 (+ 314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.969126 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.970291 (+ 1165us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.970296 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:00.971282 (+ 986us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.971313 (+ 31us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.971448 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.971472 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:00.971534 (+ 62us) client_negotiation.cc:240] Negotiation successful
1027 14:52:00.971654 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":47}
I20251027 14:52:00.972771 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.968045 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46720 (local address 127.0.104.60:43831)
1027 14:52:00.968521 (+ 476us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.968525 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.968539 (+ 14us) server_negotiation.cc:408] Connection header received
1027 14:52:00.968587 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.968591 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.968645 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.968744 (+ 99us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:00.969267 (+ 523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.970172 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.971454 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.972332 (+ 878us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.972397 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.972472 (+ 75us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.972565 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":370,"thread_start_us":233,"threads_started":1}
I20251027 14:52:00.986550 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.975126 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46728 (local address 127.0.104.60:43831)
1027 14:52:00.975260 (+ 134us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.975264 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.975307 (+ 43us) server_negotiation.cc:408] Connection header received
1027 14:52:00.975437 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.975441 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.975496 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.975572 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:00.977076 (+ 1504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.977972 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.979089 (+ 1117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.979398 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.981093 (+ 1695us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:00.981119 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:00.981122 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:00.981150 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:00.984467 (+ 3317us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:00.985075 (+ 608us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:00.985080 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:00.985082 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:00.985156 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:00.985381 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:00.985385 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:00.985387 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:00.985625 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:00.985809 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.986103 (+ 294us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.986292 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251027 14:52:00.991832 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.987640 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50194)
1027 14:52:00.988050 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:00.988065 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:00.988162 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:00.988498 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:00.988501 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:00.988523 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:00.988996 (+ 473us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.989005 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.990152 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.990156 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:00.991208 (+ 1052us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.991219 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.991350 (+ 131us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.991369 (+ 19us) client_negotiation.cc:769] Sending connection context
1027 14:52:00.991416 (+ 47us) client_negotiation.cc:240] Negotiation successful
1027 14:52:00.991510 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":215,"threads_started":1}
I20251027 14:52:00.992561 8085 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.987796 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50194 (local address 127.0.104.62:44599)
1027 14:52:00.987992 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.987999 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.988124 (+ 125us) server_negotiation.cc:408] Connection header received
1027 14:52:00.988254 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.988259 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.988334 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.988467 (+ 133us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:00.989120 (+ 653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.990004 (+ 884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.991344 (+ 1340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.992172 (+ 828us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.992218 (+ 46us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.992282 (+ 64us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.992370 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251027 14:52:00.997522 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.994048 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.0.1:51366)
1027 14:52:00.994186 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:00.994204 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:00.994286 (+ 82us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:00.994551 (+ 265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:00.994554 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:00.994589 (+ 35us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:00.994861 (+ 272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.994868 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.995990 (+ 1122us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.995994 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:00.997090 (+ 1096us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:00.997102 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.997209 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.997227 (+ 18us) client_negotiation.cc:769] Sending connection context
1027 14:52:00.997268 (+ 41us) client_negotiation.cc:240] Negotiation successful
1027 14:52:00.997346 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":37}
I20251027 14:52:00.998383 8083 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.994150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51366 (local address 127.0.104.61:37569)
1027 14:52:00.994296 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:00.994300 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:00.994333 (+ 33us) server_negotiation.cc:408] Connection header received
1027 14:52:00.994417 (+ 84us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:00.994421 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:00.994476 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:00.994539 (+ 63us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:00.994984 (+ 445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.995859 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:00.997230 (+ 1371us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:00.997989 (+ 759us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:00.998028 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:00.998087 (+ 59us) server_negotiation.cc:300] Negotiation successful
1027 14:52:00.998174 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:52:01.003715 8087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:00.999850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.0.1:46734)
1027 14:52:00.999961 (+ 111us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:00.999974 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:01.000049 (+ 75us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:01.000330 (+ 281us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:01.000334 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:01.000355 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:01.000682 (+ 327us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.000690 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.001711 (+ 1021us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.001714 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:01.003091 (+ 1377us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.003106 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.003233 (+ 127us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.003256 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:52:01.003315 (+ 59us) client_negotiation.cc:240] Negotiation successful
1027 14:52:01.003443 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":36}
I20251027 14:52:01.004690 8086 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.000015 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46734 (local address 127.0.104.60:43831)
1027 14:52:01.000144 (+ 129us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:01.000149 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:01.000162 (+ 13us) server_negotiation.cc:408] Connection header received
1027 14:52:01.000197 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:01.000200 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:01.000244 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:01.000313 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:01.000793 (+ 480us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.001578 (+ 785us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.003225 (+ 1647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.004316 (+ 1091us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.004352 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:01.004443 (+ 91us) server_negotiation.cc:300] Negotiation successful
1027 14:52:01.004525 (+ 82us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251027 14:52:01.006590 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7288
Oct 27 14:52:01 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.1@KRBTEST.COM
I20251027 14:52:01.021337 8088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.007479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36076 (local address 127.0.104.1:35535)
1027 14:52:01.007993 (+ 514us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:01.007999 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:01.008023 (+ 24us) server_negotiation.cc:408] Connection header received
1027 14:52:01.008072 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:01.008078 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:01.008271 (+ 193us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:01.008479 (+ 208us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:01.009774 (+ 1295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.010950 (+ 1176us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.011969 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.012396 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.015719 (+ 3323us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:01.015783 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:01.015798 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:01.015827 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:01.018463 (+ 2636us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.019034 (+ 571us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.019041 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.019043 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.019602 (+ 559us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.019948 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.019953 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.019955 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.020211 (+ 256us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:01.020521 (+ 310us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:01.020767 (+ 246us) server_negotiation.cc:300] Negotiation successful
1027 14:52:01.021022 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":286,"threads_started":1}
I20251027 14:52:01.022606 7411 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:01.104782 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7288
I20251027 14:52:01.118288 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:35535
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:44599
--webserver_port=38495
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:01.434154 8090 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:01.434451 8090 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:01.434552 8090 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:01.442500 8090 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:01.442597 8090 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:01.442744 8090 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:01.453147 8090 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:35535
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=38495
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:01.454635 8090 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:01.455961 8090 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:01.464994 8096 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:01.465023 8095 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:01.465552 8098 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:01.466348 8090 server_base.cc:1047] running on GCE node
I20251027 14:52:01.466889 8090 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:01.467767 8090 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:01.468997 8090 hybrid_clock.cc:648] HybridClock initialized: now 1761576721468918 us; error 55 us; skew 500 ppm
I20251027 14:52:01.469269 8090 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:01 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:01.472457 8090 init.cc:377] Logged in from keytab as oryx/127.0.104.1@KRBTEST.COM (short username oryx)
I20251027 14:52:01.473997 8090 webserver.cc:492] Webserver started at http://127.0.104.1:38495/ using document root <none> and password file <none>
I20251027 14:52:01.474702 8090 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:01.474788 8090 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:01.480144 8090 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:01.482915 8105 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:01.484163 8090 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:52:01.484287 8090 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "45af0ea8283a4637babe28d2c6441f83"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "4fa7be8cde7bdaa929b7783fe5a1ebbd"
server_key_iv: "8264a858a9c185f5953ce03a7f6704b6"
server_key_version: "encryptionkey@0"
I20251027 14:52:01.484978 8090 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:01.524463 8090 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:01.530912 8090 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:01.531149 8090 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:01.532032 8090 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:01.533540 8090 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:01.533610 8090 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:01.533696 8090 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:01.533740 8090 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:01.563711 8090 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35535
I20251027 14:52:01.563755 8217 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35535 every 8 connection(s)
I20251027 14:52:01.565142 8090 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:01.568114 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 8090
I20251027 14:52:01.568462 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7424
Oct 27 14:52:01 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
I20251027 14:52:01.592939 8222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.568234 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:47071 (local address 127.0.104.61:37569)
1027 14:52:01.568727 (+ 493us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:01.568732 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:01.570556 (+ 1824us) server_negotiation.cc:408] Connection header received
1027 14:52:01.572958 (+ 2402us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:01.572966 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:01.573060 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:01.573149 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:01.576151 (+ 3002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.577643 (+ 1492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.579620 (+ 1977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.580073 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.585037 (+ 4964us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:01.585098 (+ 61us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:01.585104 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:01.585148 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:01.589050 (+ 3902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.590142 (+ 1092us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.590149 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.590153 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.590245 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.590669 (+ 424us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.590674 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.590678 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.590988 (+ 310us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:01.591110 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:01.592274 (+ 1164us) server_negotiation.cc:300] Negotiation successful
1027 14:52:01.592574 (+ 300us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":400,"thread_start_us":281,"threads_started":1}
I20251027 14:52:01.595826 8226 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.568974 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.104.1:47071)
1027 14:52:01.570373 (+ 1399us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:01.570432 (+ 59us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:01.572560 (+ 2128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:01.573834 (+ 1274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:01.573848 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:01.574745 (+ 897us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:01.575910 (+ 1165us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.575940 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.577871 (+ 1931us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.577877 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:01.579419 (+ 1542us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.579439 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.580455 (+ 1016us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.581318 (+ 863us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:01.581354 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:01.584767 (+ 3413us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:01.589319 (+ 4552us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:01.589329 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:01.589350 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:01.589991 (+ 641us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:01.590378 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:01.590384 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:01.590388 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:01.590542 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:01.591217 (+ 675us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:01.591226 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:01.591736 (+ 510us) client_negotiation.cc:769] Sending connection context
1027 14:52:01.593573 (+ 1837us) client_negotiation.cc:240] Negotiation successful
1027 14:52:01.594146 (+ 573us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":10256,"threads_started":1}
Oct 27 14:52:01 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.2@KRBTEST.COM
I20251027 14:52:01.598165 8220 heartbeater.cc:344] Connected to a master server at 127.0.104.61:37569
I20251027 14:52:01.598595 8220 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:01.599592 8220 heartbeater.cc:507] Master 127.0.104.61:37569 requested a full tablet report, sending...
I20251027 14:52:01.601734 7834 ts_manager.cc:194] Registered new tserver with Master: 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535)
I20251027 14:52:01.606238 8225 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.569361 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35034 (local address 127.0.104.2:46289)
1027 14:52:01.584091 (+ 14730us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:01.584103 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:01.584141 (+ 38us) server_negotiation.cc:408] Connection header received
1027 14:52:01.584222 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:01.584232 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:01.584560 (+ 328us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:01.584786 (+ 226us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:01.587606 (+ 2820us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.589734 (+ 2128us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.591715 (+ 1981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.593032 (+ 1317us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.597246 (+ 4214us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:01.597331 (+ 85us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:01.597351 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:01.597405 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:01.601838 (+ 4433us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.602787 (+ 949us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.602799 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.602803 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.602923 (+ 120us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.603253 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.603258 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.603262 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.603618 (+ 356us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:01.603928 (+ 310us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:01.604220 (+ 292us) server_negotiation.cc:300] Negotiation successful
1027 14:52:01.605946 (+ 1726us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13465,"thread_start_us":14326,"threads_started":1}
Oct 27 14:52:01 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:01.607602 7547 generic_service.cc:196] Checking for leaks (request via RPC)
Oct 27 14:52:01 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576721, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.60@KRBTEST.COM
I20251027 14:52:01.613699 8229 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.579845 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:44479)
1027 14:52:01.592430 (+ 12585us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:01.592451 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:01.592566 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:01.595281 (+ 2715us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:01.595287 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:01.595883 (+ 596us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:01.596478 (+ 595us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.596501 (+ 23us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.602998 (+ 6497us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.603004 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:01.604009 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.604024 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.604165 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.604888 (+ 723us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:01.604925 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:01.607509 (+ 2584us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:01.611131 (+ 3622us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:01.611137 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:01.611142 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:01.611702 (+ 560us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:01.612091 (+ 389us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:01.612100 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:01.612104 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:01.612185 (+ 81us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:01.612734 (+ 549us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:01.612739 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:01.612869 (+ 130us) client_negotiation.cc:769] Sending connection context
1027 14:52:01.612994 (+ 125us) client_negotiation.cc:240] Negotiation successful
1027 14:52:01.613261 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12441,"thread_start_us":12267,"threads_started":1}
I20251027 14:52:01.614634 8228 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.579833 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:44479 (local address 127.0.104.62:44599)
1027 14:52:01.594913 (+ 15080us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:01.594920 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:01.594938 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:52:01.594995 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:01.595002 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:01.595081 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:01.596665 (+ 1584us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:01.596877 (+ 212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.602816 (+ 5939us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.604547 (+ 1731us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.604929 (+ 382us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.607745 (+ 2816us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:01.607784 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:01.607789 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:01.607832 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:01.611007 (+ 3175us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.611856 (+ 849us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.611863 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.611867 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.611965 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.612311 (+ 346us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.612315 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.612317 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.612617 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:01.612758 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:01.613150 (+ 392us) server_negotiation.cc:300] Negotiation successful
1027 14:52:01.613411 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14962,"thread_start_us":14887,"threads_started":1}
I20251027 14:52:01.615450 8219 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44599
I20251027 14:52:01.615620 8219 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:01.616107 8219 heartbeater.cc:507] Master 127.0.104.62:44599 requested a full tablet report, sending...
I20251027 14:52:01.617408 7224 ts_manager.cc:194] Re-registered known tserver with Master: 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535)
I20251027 14:52:01.618554 7224 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:44479
I20251027 14:52:01.619558 8224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.569645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.104.1:34305)
1027 14:52:01.579589 (+ 9944us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:01.579607 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:01.579726 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:01.605625 (+ 25899us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:01.605631 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:01.606220 (+ 589us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:01.606719 (+ 499us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.606731 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.609533 (+ 2802us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.609539 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:01.610772 (+ 1233us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:01.610787 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.610927 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.611491 (+ 564us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:01.611527 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:01.614219 (+ 2692us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:01.617628 (+ 3409us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:01.617633 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:01.617638 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:01.617932 (+ 294us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:01.618224 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:01.618228 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:01.618231 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:01.618304 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:01.618737 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:01.618741 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:01.618841 (+ 100us) client_negotiation.cc:769] Sending connection context
1027 14:52:01.618949 (+ 108us) client_negotiation.cc:240] Negotiation successful
1027 14:52:01.619199 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9824,"thread_start_us":382,"threads_started":1}
I20251027 14:52:01.620359 8227 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:01.570160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:34305 (local address 127.0.104.60:43831)
1027 14:52:01.605319 (+ 35159us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:01.605324 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:01.605339 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:52:01.605384 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:01.605389 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:01.605456 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:01.605573 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:01.606894 (+ 1321us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.609341 (+ 2447us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:01.611486 (+ 2145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:01.611993 (+ 507us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:01.614420 (+ 2427us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:01.614454 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:01.614458 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:01.614489 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:01.617506 (+ 3017us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.618037 (+ 531us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.618043 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.618046 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.618125 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:01.618394 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:01.618399 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:01.618403 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:01.618612 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:01.619886 (+ 1274us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:01.619984 (+ 98us) server_negotiation.cc:300] Negotiation successful
1027 14:52:01.620152 (+ 168us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35059,"thread_start_us":34951,"threads_started":1}
I20251027 14:52:01.621229 8218 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43831
I20251027 14:52:01.621351 8218 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:01.621677 8218 heartbeater.cc:507] Master 127.0.104.60:43831 requested a full tablet report, sending...
I20251027 14:52:01.623296 8022 ts_manager.cc:194] Registered new tserver with Master: 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535)
I20251027 14:52:01.691643 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7424
I20251027 14:52:01.705031 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:46289
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:44599
--webserver_port=36459
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:02.034598 8231 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:02.034909 8231 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:02.035043 8231 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:02.043156 8231 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:02.043273 8231 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:02.043462 8231 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:02.053936 8231 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:46289
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=36459
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:02.055509 8231 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:02.056816 8231 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:02.065549 8236 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:02.065613 8237 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:02.066157 8239 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:02.066838 8231 server_base.cc:1047] running on GCE node
I20251027 14:52:02.067425 8231 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:02.068553 8231 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:02.069782 8231 hybrid_clock.cc:648] HybridClock initialized: now 1761576722069687 us; error 69 us; skew 500 ppm
I20251027 14:52:02.070089 8231 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:02.073480 8231 init.cc:377] Logged in from keytab as oryx/127.0.104.2@KRBTEST.COM (short username oryx)
I20251027 14:52:02.075075 8231 webserver.cc:492] Webserver started at http://127.0.104.2:36459/ using document root <none> and password file <none>
I20251027 14:52:02.075799 8231 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:02.075887 8231 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:02.081634 8231 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20251027 14:52:02.084487 8246 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:02.085840 8231 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:52:02.085990 8231 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "69e31b5ad78c4358b932f61f8dc1d662"
format_stamp: "Formatted at 2025-10-27 14:51:56 on dist-test-slave-hxxt"
server_key: "13ec69f77de5795402e4a2eb2a2dd1b2"
server_key_iv: "dbf57f4d07308a55e523a888469a39e3"
server_key_version: "encryptionkey@0"
I20251027 14:52:02.086714 8231 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:02.110363 8231 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:02.114364 8231 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:02.114603 8231 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:02.115479 8231 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:02.117015 8231 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:02.117089 8231 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:02.117183 8231 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:02.117241 8231 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:02.146543 8231 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:46289
I20251027 14:52:02.146584 8358 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:46289 every 8 connection(s)
I20251027 14:52:02.147940 8231 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:02.154825 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 8231
I20251027 14:52:02.155088 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7559
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.60@KRBTEST.COM
I20251027 14:52:02.173034 8364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.150975 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:49999 (local address 127.0.104.61:37569)
1027 14:52:02.151724 (+ 749us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.151730 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.152668 (+ 938us) server_negotiation.cc:408] Connection header received
1027 14:52:02.154634 (+ 1966us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.154640 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.154723 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.154791 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.158242 (+ 3451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.159171 (+ 929us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.160871 (+ 1700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.161301 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.165766 (+ 4465us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.165802 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.165807 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.165846 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.169627 (+ 3781us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.170612 (+ 985us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.170618 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.170622 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.170708 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.171113 (+ 405us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.171118 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.171122 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.171425 (+ 303us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.171571 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.172455 (+ 884us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.172677 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":580,"thread_start_us":486,"threads_started":1}
I20251027 14:52:02.174508 8366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.151741 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.104.2:49999)
1027 14:52:02.152481 (+ 740us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.152539 (+ 58us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.154290 (+ 1751us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.155222 (+ 932us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.155231 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.157051 (+ 1820us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:02.158056 (+ 1005us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.158082 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.159299 (+ 1217us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.159304 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.160701 (+ 1397us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.160716 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.161014 (+ 298us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.161816 (+ 802us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:02.161852 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:02.165009 (+ 3157us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:02.169799 (+ 4790us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.169809 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.169829 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.170480 (+ 651us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.170821 (+ 341us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.170827 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.170831 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.170996 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.171591 (+ 595us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:02.171597 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:02.171941 (+ 344us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.172269 (+ 328us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.172836 (+ 567us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":593,"thread_start_us":403,"threads_started":1}
I20251027 14:52:02.174989 8365 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.151562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.104.2:42421)
1027 14:52:02.152490 (+ 928us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.152540 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.154300 (+ 1760us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.155224 (+ 924us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.155232 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.158454 (+ 3222us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:02.159040 (+ 586us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.159052 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.160248 (+ 1196us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.160252 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.161344 (+ 1092us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.161357 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.161565 (+ 208us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.163099 (+ 1534us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:02.163163 (+ 64us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:02.168223 (+ 5060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:02.172433 (+ 4210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.172438 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.172442 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.172769 (+ 327us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.173105 (+ 336us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.173110 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.173114 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.173205 (+ 91us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.173836 (+ 631us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:02.173842 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:02.173978 (+ 136us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.174109 (+ 131us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.174371 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"spinlock_wait_cycles":28928,"thread_start_us":352,"threads_started":1}
I20251027 14:52:02.175771 8363 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.150889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:42421 (local address 127.0.104.60:43831)
1027 14:52:02.151349 (+ 460us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.151356 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.152665 (+ 1309us) server_negotiation.cc:408] Connection header received
1027 14:52:02.154637 (+ 1972us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.154643 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.154725 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.154811 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.159207 (+ 4396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.160114 (+ 907us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.161528 (+ 1414us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.161951 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.168513 (+ 6562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.168562 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.168568 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.168618 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.172249 (+ 3631us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.172892 (+ 643us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.172898 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.172902 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.172981 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.173339 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.173345 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.173349 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.173625 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.173775 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.174678 (+ 903us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.174919 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":307,"thread_start_us":717,"threads_started":1}
I20251027 14:52:02.176796 8361 heartbeater.cc:344] Connected to a master server at 127.0.104.61:37569
I20251027 14:52:02.176831 8359 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43831
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576715, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.3@KRBTEST.COM
I20251027 14:52:02.177260 8361 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:02.178308 8361 heartbeater.cc:507] Master 127.0.104.61:37569 requested a full tablet report, sending...
I20251027 14:52:02.177264 8359 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:02.179433 8359 heartbeater.cc:507] Master 127.0.104.60:43831 requested a full tablet report, sending...
I20251027 14:52:02.180199 7834 ts_manager.cc:194] Registered new tserver with Master: 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289)
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:02.180828 8022 ts_manager.cc:194] Registered new tserver with Master: 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289)
I20251027 14:52:02.185268 8370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.156068 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44180 (local address 127.0.104.3:45563)
1027 14:52:02.161452 (+ 5384us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.161461 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.161500 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:52:02.161565 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.161574 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.162509 (+ 935us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.162694 (+ 185us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.165130 (+ 2436us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.167353 (+ 2223us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.170518 (+ 3165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.170961 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.177970 (+ 7009us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.178038 (+ 68us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.178051 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.178089 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.182251 (+ 4162us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.182970 (+ 719us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.182978 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.182981 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.183063 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.183356 (+ 293us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.183360 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.183362 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.183638 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.183925 (+ 287us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.184827 (+ 902us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.185026 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":449,"thread_start_us":325,"threads_started":1}
I20251027 14:52:02.186239 8367 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.153079 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:56821)
1027 14:52:02.153664 (+ 585us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.153681 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.154301 (+ 620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.170212 (+ 15911us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.170218 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.171668 (+ 1450us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:02.172290 (+ 622us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.172302 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.175382 (+ 3080us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.175388 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.177051 (+ 1663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.177064 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.177258 (+ 194us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.178225 (+ 967us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:02.178244 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:02.181276 (+ 3032us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:02.184089 (+ 2813us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.184095 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.184098 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.184550 (+ 452us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.184873 (+ 323us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.184878 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.184882 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.184957 (+ 75us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.185402 (+ 445us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:02.185407 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:02.185536 (+ 129us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.185649 (+ 113us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.185901 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":262,"threads_started":1}
I20251027 14:52:02.186704 7682 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:02.186980 8368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.153206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:56821 (local address 127.0.104.62:44599)
1027 14:52:02.169835 (+ 16629us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.169842 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.169887 (+ 45us) server_negotiation.cc:408] Connection header received
1027 14:52:02.169946 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.169952 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.170030 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.170145 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.173281 (+ 3136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.174600 (+ 1319us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.177558 (+ 2958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.177993 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.181425 (+ 3432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.181453 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.181457 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.181489 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.183970 (+ 2481us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.184663 (+ 693us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.184669 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.184672 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.184770 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.185058 (+ 288us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.185063 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.185065 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.185290 (+ 225us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.186418 (+ 1128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.186535 (+ 117us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.186726 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16489,"thread_start_us":17080,"threads_started":1}
I20251027 14:52:02.187712 8360 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44599
I20251027 14:52:02.187865 8360 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:02.188432 8360 heartbeater.cc:507] Master 127.0.104.62:44599 requested a full tablet report, sending...
I20251027 14:52:02.190086 7224 ts_manager.cc:194] Re-registered known tserver with Master: 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289)
I20251027 14:52:02.191146 7224 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:56821
I20251027 14:52:02.268599 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7559
I20251027 14:52:02.282186 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:45563
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:44599
--webserver_port=45311
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39825
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:02.604353 8372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:02.604694 8372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:02.604827 8372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:02.612877 8372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:02.612978 8372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:02.613162 8372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:02.621544 8219 heartbeater.cc:499] Master 127.0.104.62:44599 was elected leader, sending a full tablet report...
I20251027 14:52:02.623629 8372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39825
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:45563
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=45311
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:44599,127.0.104.61:37569,127.0.104.60:43831
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:02.625348 8372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:02.626641 8372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:02.635398 8378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:02.635695 8377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:02.636438 8380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:02.636708 8372 server_base.cc:1047] running on GCE node
I20251027 14:52:02.637279 8372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:02.638233 8372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:02.639459 8372 hybrid_clock.cc:648] HybridClock initialized: now 1761576722639401 us; error 43 us; skew 500 ppm
I20251027 14:52:02.639725 8372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:02.642763 8372 init.cc:377] Logged in from keytab as oryx/127.0.104.3@KRBTEST.COM (short username oryx)
I20251027 14:52:02.644176 8372 webserver.cc:492] Webserver started at http://127.0.104.3:45311/ using document root <none> and password file <none>
I20251027 14:52:02.644905 8372 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:02.644991 8372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:02.650362 8372 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251027 14:52:02.653340 8387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:02.654691 8372 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:02.654847 8372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "af3b3966dd764458af1f02e364788f7a"
format_stamp: "Formatted at 2025-10-27 14:51:57 on dist-test-slave-hxxt"
server_key: "9241d96096bdb631cfd44b92139b6410"
server_key_iv: "8e7e06292a909bc115a7e3774ffc502a"
server_key_version: "encryptionkey@0"
I20251027 14:52:02.655510 8372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:02.690637 8372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:02.694514 8372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:02.694777 8372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:02.695706 8372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:02.697211 8372 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:02.697296 8372 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:02.697410 8372 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:02.697463 8372 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:02.727147 8372 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45563
I20251027 14:52:02.727211 8500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45563 every 8 connection(s)
I20251027 14:52:02.728638 8372 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:02.732102 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 8372
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.60@KRBTEST.COM
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:02.787060 8505 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.731608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:43919 (local address 127.0.104.60:43831)
1027 14:52:02.744362 (+ 12754us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.744369 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.752796 (+ 8427us) server_negotiation.cc:408] Connection header received
1027 14:52:02.755123 (+ 2327us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.755133 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.755223 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.755312 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.762317 (+ 7005us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.763991 (+ 1674us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.768960 (+ 4969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.769423 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.775172 (+ 5749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.775209 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.775225 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.775262 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.783853 (+ 8591us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.784736 (+ 883us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.784742 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.784746 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.784828 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.785199 (+ 371us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.785204 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.785208 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.785510 (+ 302us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.785640 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.786488 (+ 848us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.786725 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12632,"thread_start_us":12526,"threads_started":1}
I20251027 14:52:02.787995 8513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.746360 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50204 (local address 127.0.104.62:44599)
1027 14:52:02.770533 (+ 24173us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.770540 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.770559 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:52:02.770618 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.770624 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.770700 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.770824 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.773374 (+ 2550us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.775939 (+ 2565us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.777879 (+ 1940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.778354 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.782252 (+ 3898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.782309 (+ 57us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.782314 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.782354 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.785492 (+ 3138us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.786223 (+ 731us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.786227 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.786230 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.786292 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.786595 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.786599 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.786601 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.786804 (+ 203us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.787020 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.787517 (+ 497us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.787744 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":24020,"thread_start_us":24633,"threads_started":1}
I20251027 14:52:02.788337 8507 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.732888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43831 (local address 127.0.104.3:43919)
1027 14:52:02.752696 (+ 19808us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.752715 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.754828 (+ 2113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.756079 (+ 1251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.756093 (+ 14us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.760802 (+ 4709us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:02.761336 (+ 534us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.761348 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.764267 (+ 2919us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.764273 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.765914 (+ 1641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.765931 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.769136 (+ 3205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.771424 (+ 2288us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:02.771475 (+ 51us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:02.774930 (+ 3455us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:02.783994 (+ 9064us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.784004 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.784025 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.784624 (+ 599us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.784923 (+ 299us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.784927 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.784931 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.785092 (+ 161us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.785613 (+ 521us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:02.785622 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:02.786011 (+ 389us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.786368 (+ 357us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.786847 (+ 479us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19681,"thread_start_us":11678,"threads_started":1}
I20251027 14:52:02.789983 8501 heartbeater.cc:344] Connected to a master server at 127.0.104.60:43831
I20251027 14:52:02.790344 8501 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:02.791157 8501 heartbeater.cc:507] Master 127.0.104.60:43831 requested a full tablet report, sending...
I20251027 14:52:02.791261 8506 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.731699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:48153 (local address 127.0.104.62:44599)
1027 14:52:02.743948 (+ 12249us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.743955 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.745108 (+ 1153us) server_negotiation.cc:408] Connection header received
1027 14:52:02.757447 (+ 12339us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.757455 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.757553 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.757644 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.760095 (+ 2451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.761484 (+ 1389us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.771228 (+ 9744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.771639 (+ 411us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.778988 (+ 7349us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.779023 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.779027 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.779064 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.782975 (+ 3911us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.789538 (+ 6563us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.789544 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.789547 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.789630 (+ 83us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.789914 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.789919 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.789923 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.790167 (+ 244us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.790287 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.790688 (+ 401us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.790894 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12106,"thread_start_us":12256,"threads_started":1}
I20251027 14:52:02.791260 8508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.732538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:48153)
1027 14:52:02.744916 (+ 12378us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.744974 (+ 58us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.754021 (+ 9047us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.757929 (+ 3908us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.757934 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.758830 (+ 896us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:02.759913 (+ 1083us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.759940 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.766987 (+ 7047us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.766994 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.768308 (+ 1314us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.768324 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.771308 (+ 2984us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.773047 (+ 1739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:02.773069 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:02.778845 (+ 5776us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:02.788758 (+ 9913us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.788763 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.788767 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.789427 (+ 660us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.789729 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.789734 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.789738 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.789812 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.790277 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:02.790281 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:02.790427 (+ 146us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.790566 (+ 139us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.790848 (+ 282us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12092,"thread_start_us":19772,"threads_started":1}
I20251027 14:52:02.793109 8022 ts_manager.cc:194] Registered new tserver with Master: af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563)
I20251027 14:52:02.793279 8502 heartbeater.cc:344] Connected to a master server at 127.0.104.62:44599
I20251027 14:52:02.793413 8502 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:02.793998 8502 heartbeater.cc:507] Master 127.0.104.62:44599 requested a full tablet report, sending...
Oct 27 14:52:02 dist-test-slave-hxxt krb5kdc[7177](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576722, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.61@KRBTEST.COM
I20251027 14:52:02.794888 7224 ts_manager.cc:194] Re-registered known tserver with Master: af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563)
I20251027 14:52:02.795919 7224 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:48153
I20251027 14:52:02.798274 7224 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50204:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:52:02.803761 7224 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:02.804627 8520 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.761732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51382 (local address 127.0.104.61:37569)
1027 14:52:02.773234 (+ 11502us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.773240 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.773258 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:52:02.776085 (+ 2827us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.776091 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.776179 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.776268 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.783064 (+ 6796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.788601 (+ 5537us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.796230 (+ 7629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.796743 (+ 513us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.799217 (+ 2474us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.799253 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.799258 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.799289 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.802272 (+ 2983us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.802878 (+ 606us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.802882 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.802884 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.802949 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.803247 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.803252 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.803255 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.803470 (+ 215us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.803698 (+ 228us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.804021 (+ 323us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.804256 (+ 235us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11388,"thread_start_us":14736,"threads_started":1}
I20251027 14:52:02.807525 8516 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.746889 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46744 (local address 127.0.104.60:43831)
1027 14:52:02.748818 (+ 1929us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.748825 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.766212 (+ 17387us) server_negotiation.cc:408] Connection header received
1027 14:52:02.767262 (+ 1050us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.767269 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.767349 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.767419 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.770248 (+ 2829us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.773881 (+ 3633us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.784017 (+ 10136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.784510 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.797669 (+ 13159us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.797715 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.797720 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.797757 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.801409 (+ 3652us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.805120 (+ 3711us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.805127 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.805129 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.805215 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.806182 (+ 967us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.806188 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.806193 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.806519 (+ 326us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.806788 (+ 269us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.807058 (+ 270us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.807236 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1808,"thread_start_us":1702,"threads_started":1}
I20251027 14:52:02.810770 8518 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.753003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:46513 (local address 127.0.104.61:37569)
1027 14:52:02.770450 (+ 17447us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.770456 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.770484 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:52:02.770540 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.770546 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.770635 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.770736 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:02.774071 (+ 3335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.775422 (+ 1351us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.784945 (+ 9523us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.785949 (+ 1004us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.805056 (+ 19107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:02.805125 (+ 69us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:02.805131 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:02.805163 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:02.808483 (+ 3320us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.809065 (+ 582us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.809069 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.809071 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.809129 (+ 58us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:02.809424 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:02.809427 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:02.809428 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:02.809665 (+ 237us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:02.809805 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.810161 (+ 356us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.810380 (+ 219us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17331,"thread_start_us":17918,"threads_started":1}
I20251027 14:52:02.810894 8517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.752889 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:37569 (local address 127.0.104.3:46513)
1027 14:52:02.759406 (+ 6517us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.759425 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.759540 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.771143 (+ 11603us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.771148 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.772220 (+ 1072us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:02.772820 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.772838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.776733 (+ 3895us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.776739 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.779305 (+ 2566us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.779315 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.779450 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.789444 (+ 9994us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:02.789468 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:02.795324 (+ 5856us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:02.808611 (+ 13287us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.808615 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.808618 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.808950 (+ 332us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.809226 (+ 276us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:02.809231 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:02.809234 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:02.809335 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:02.809772 (+ 437us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:02.809777 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:02.809915 (+ 138us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.810036 (+ 121us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.810531 (+ 495us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":6405,"thread_start_us":6720,"threads_started":1}
I20251027 14:52:02.811708 8503 heartbeater.cc:344] Connected to a master server at 127.0.104.61:37569
I20251027 14:52:02.811865 8503 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:02.812294 8503 heartbeater.cc:507] Master 127.0.104.61:37569 requested a full tablet report, sending...
I20251027 14:52:02.820166 7834 ts_manager.cc:194] Registered new tserver with Master: af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563)
I20251027 14:52:02.855790 8535 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.830902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:46289 (local address 127.0.0.1:35038)
1027 14:52:02.844832 (+ 13930us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.844852 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.845580 (+ 728us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.848547 (+ 2967us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.848554 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.848587 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:02.850859 (+ 2272us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.850874 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.853206 (+ 2332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.853211 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.855240 (+ 2029us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.855251 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.855381 (+ 130us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.855402 (+ 21us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.855454 (+ 52us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.855550 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13774,"thread_start_us":20186,"threads_started":1}
I20251027 14:52:02.857093 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.831185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35038 (local address 127.0.104.2:46289)
1027 14:52:02.843736 (+ 12551us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.843747 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.846440 (+ 2693us) server_negotiation.cc:408] Connection header received
1027 14:52:02.846586 (+ 146us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.846596 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.847313 (+ 717us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.847561 (+ 248us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:02.851070 (+ 3509us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.853069 (+ 1999us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.855370 (+ 2301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.856430 (+ 1060us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.856587 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.856680 (+ 93us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.856839 (+ 159us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12339,"thread_start_us":15010,"threads_started":1}
I20251027 14:52:02.857442 8531 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.834651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:45563 (local address 127.0.0.1:44188)
1027 14:52:02.851353 (+ 16702us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.851369 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.851468 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.851942 (+ 474us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.851947 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.851968 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:02.852535 (+ 567us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.852545 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.855073 (+ 2528us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.855077 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.856774 (+ 1697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.856791 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.856962 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.856986 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.857048 (+ 62us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.857157 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16590,"thread_start_us":9926,"threads_started":1}
I20251027 14:52:02.857767 8533 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.831503 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:35535 (local address 127.0.0.1:36084)
1027 14:52:02.848428 (+ 16925us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.848448 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.848576 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.850045 (+ 1469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.850048 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.850068 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:02.850733 (+ 665us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.850744 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.852870 (+ 2126us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.852875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.857099 (+ 4224us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.857112 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.857279 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.857303 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.857363 (+ 60us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.857471 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":16775,"thread_start_us":16630,"threads_started":1}
I20251027 14:52:02.858991 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.831673 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.0.104.1:35535)
1027 14:52:02.849409 (+ 17736us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.849419 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.849450 (+ 31us) server_negotiation.cc:408] Connection header received
1027 14:52:02.849574 (+ 124us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.849584 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.849850 (+ 266us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.850068 (+ 218us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:02.850885 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.852705 (+ 1820us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.857638 (+ 4933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.858520 (+ 882us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.858624 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.858701 (+ 77us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.858809 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17563,"thread_start_us":17444,"threads_started":1}
I20251027 14:52:02.859305 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.834953 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44188 (local address 127.0.104.3:45563)
1027 14:52:02.846172 (+ 11219us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.846183 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.851465 (+ 5282us) server_negotiation.cc:408] Connection header received
1027 14:52:02.851610 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.851617 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.851782 (+ 165us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.851945 (+ 163us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:02.852683 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.854888 (+ 2205us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.856929 (+ 2041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.858660 (+ 1731us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.858784 (+ 124us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.858908 (+ 124us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.859052 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11018,"thread_start_us":10868,"threads_started":1}
I20251027 14:52:02.861888 8294 tablet_service.cc:1505] Processing CreateTablet for tablet bee08dbe6d3a41898ed48c841e88b015 (DEFAULT_TABLE table=test-table [id=4cc5544649fd4239a9421966b0b76e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:02.862434 8436 tablet_service.cc:1505] Processing CreateTablet for tablet bee08dbe6d3a41898ed48c841e88b015 (DEFAULT_TABLE table=test-table [id=4cc5544649fd4239a9421966b0b76e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:02.863337 8294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee08dbe6d3a41898ed48c841e88b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:02.863083 8153 tablet_service.cc:1505] Processing CreateTablet for tablet bee08dbe6d3a41898ed48c841e88b015 (DEFAULT_TABLE table=test-table [id=4cc5544649fd4239a9421966b0b76e2d]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:02.864025 8436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee08dbe6d3a41898ed48c841e88b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:02.864346 8153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bee08dbe6d3a41898ed48c841e88b015. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:02.874792 8537 tablet_bootstrap.cc:492] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Bootstrap starting.
I20251027 14:52:02.877586 8538 tablet_bootstrap.cc:492] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Bootstrap starting.
I20251027 14:52:02.878021 8537 tablet_bootstrap.cc:654] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:02.879317 8537 log.cc:826] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:02.879963 8539 tablet_bootstrap.cc:492] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Bootstrap starting.
I20251027 14:52:02.881733 8538 tablet_bootstrap.cc:654] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:02.883069 8538 log.cc:826] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:02.883186 8539 tablet_bootstrap.cc:654] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:02.884276 8539 log.cc:826] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:02.886391 8538 tablet_bootstrap.cc:492] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: No bootstrap required, opened a new log
I20251027 14:52:02.886611 8539 tablet_bootstrap.cc:492] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: No bootstrap required, opened a new log
I20251027 14:52:02.886703 8538 ts_tablet_manager.cc:1403] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20251027 14:52:02.886870 8539 ts_tablet_manager.cc:1403] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Time spent bootstrapping tablet: real 0.007s user 0.005s sys 0.000s
I20251027 14:52:02.889926 8537 tablet_bootstrap.cc:492] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: No bootstrap required, opened a new log
I20251027 14:52:02.890190 8537 ts_tablet_manager.cc:1403] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Time spent bootstrapping tablet: real 0.016s user 0.006s sys 0.006s
I20251027 14:52:02.892395 8539 raft_consensus.cc:359] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.892776 8539 raft_consensus.cc:385] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:02.892881 8539 raft_consensus.cc:740] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45af0ea8283a4637babe28d2c6441f83, State: Initialized, Role: FOLLOWER
I20251027 14:52:02.893118 8538 raft_consensus.cc:359] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.893455 8538 raft_consensus.cc:385] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:02.893560 8538 raft_consensus.cc:740] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e31b5ad78c4358b932f61f8dc1d662, State: Initialized, Role: FOLLOWER
I20251027 14:52:02.893492 8539 consensus_queue.cc:260] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.894134 8538 consensus_queue.cc:260] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.895185 8360 heartbeater.cc:499] Master 127.0.104.62:44599 was elected leader, sending a full tablet report...
I20251027 14:52:02.895772 8538 ts_tablet_manager.cc:1434] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Time spent starting tablet: real 0.009s user 0.010s sys 0.000s
I20251027 14:52:02.897562 8539 ts_tablet_manager.cc:1434] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Time spent starting tablet: real 0.011s user 0.006s sys 0.004s
W20251027 14:52:02.900606 8362 tablet.cc:2378] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:02.903869 8537 raft_consensus.cc:359] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.904161 8537 raft_consensus.cc:385] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:02.904233 8537 raft_consensus.cc:740] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b3966dd764458af1f02e364788f7a, State: Initialized, Role: FOLLOWER
I20251027 14:52:02.904803 8537 consensus_queue.cc:260] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.905740 8502 heartbeater.cc:499] Master 127.0.104.62:44599 was elected leader, sending a full tablet report...
I20251027 14:52:02.906294 8537 ts_tablet_manager.cc:1434] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Time spent starting tablet: real 0.016s user 0.010s sys 0.000s
I20251027 14:52:02.963798 8544 raft_consensus.cc:493] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:02.964094 8544 raft_consensus.cc:515] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.966101 8544 leader_election.cc:290] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535), af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563)
I20251027 14:52:02.972525 8547 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.966872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:35535 (local address 127.0.104.2:50279)
1027 14:52:02.967622 (+ 750us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.967635 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.967774 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.968106 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.968112 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.968138 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:02.968940 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.968952 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.970577 (+ 1625us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.970583 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.971735 (+ 1152us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.971747 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.971917 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.971944 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.972018 (+ 74us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.972132 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"spinlock_wait_cycles":49280,"thread_start_us":538,"threads_started":1}
I20251027 14:52:02.973308 8534 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.966837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:50279 (local address 127.0.104.1:35535)
1027 14:52:02.967067 (+ 230us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.967073 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.967697 (+ 624us) server_negotiation.cc:408] Connection header received
1027 14:52:02.967945 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.967950 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.968004 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.968074 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:02.969089 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.970366 (+ 1277us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.971886 (+ 1520us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.972924 (+ 1038us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.972974 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.973055 (+ 81us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.973145 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":116}
I20251027 14:52:02.974318 8536 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.966837 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:55513 (local address 127.0.104.3:45563)
1027 14:52:02.967110 (+ 273us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:02.967116 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:02.967665 (+ 549us) server_negotiation.cc:408] Connection header received
1027 14:52:02.967918 (+ 253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:02.967926 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:02.968003 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:02.968090 (+ 87us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:02.969300 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.970491 (+ 1191us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.972608 (+ 2117us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.973665 (+ 1057us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.973709 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:02.974028 (+ 319us) server_negotiation.cc:300] Negotiation successful
1027 14:52:02.974132 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":167}
I20251027 14:52:02.974347 8546 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:02.966339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:45563 (local address 127.0.104.2:55513)
1027 14:52:02.967558 (+ 1219us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:02.967580 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:02.967772 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:02.968299 (+ 527us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:02.968305 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:02.968394 (+ 89us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:02.969147 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.969160 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.970686 (+ 1526us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:02.970692 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:02.972444 (+ 1752us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:02.972457 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:02.973827 (+ 1370us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:02.973859 (+ 32us) client_negotiation.cc:769] Sending connection context
1027 14:52:02.973977 (+ 118us) client_negotiation.cc:240] Negotiation successful
1027 14:52:02.974085 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1091,"spinlock_wait_cycles":768,"thread_start_us":883,"threads_started":1}
I20251027 14:52:02.974838 8173 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bee08dbe6d3a41898ed48c841e88b015" candidate_uuid: "69e31b5ad78c4358b932f61f8dc1d662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45af0ea8283a4637babe28d2c6441f83" is_pre_election: true
I20251027 14:52:02.975334 8173 raft_consensus.cc:2468] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e31b5ad78c4358b932f61f8dc1d662 in term 0.
I20251027 14:52:02.975317 8456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bee08dbe6d3a41898ed48c841e88b015" candidate_uuid: "69e31b5ad78c4358b932f61f8dc1d662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b3966dd764458af1f02e364788f7a" is_pre_election: true
I20251027 14:52:02.975751 8456 raft_consensus.cc:2468] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 69e31b5ad78c4358b932f61f8dc1d662 in term 0.
I20251027 14:52:02.975978 8247 leader_election.cc:304] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45af0ea8283a4637babe28d2c6441f83, 69e31b5ad78c4358b932f61f8dc1d662; no voters:
I20251027 14:52:02.976368 8544 raft_consensus.cc:2804] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:02.976539 8544 raft_consensus.cc:493] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:02.976637 8544 raft_consensus.cc:3060] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:02.978919 8544 raft_consensus.cc:515] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.979516 8544 leader_election.cc:290] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [CANDIDATE]: Term 1 election: Requested vote from peers 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535), af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563)
I20251027 14:52:02.979888 8173 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bee08dbe6d3a41898ed48c841e88b015" candidate_uuid: "69e31b5ad78c4358b932f61f8dc1d662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "45af0ea8283a4637babe28d2c6441f83"
I20251027 14:52:02.979990 8456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bee08dbe6d3a41898ed48c841e88b015" candidate_uuid: "69e31b5ad78c4358b932f61f8dc1d662" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b3966dd764458af1f02e364788f7a"
I20251027 14:52:02.980051 8173 raft_consensus.cc:3060] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:02.980166 8456 raft_consensus.cc:3060] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Advancing to term 1
W20251027 14:52:02.981408 8504 tablet.cc:2378] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:02.982470 8173 raft_consensus.cc:2468] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e31b5ad78c4358b932f61f8dc1d662 in term 1.
I20251027 14:52:02.982869 8247 leader_election.cc:304] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45af0ea8283a4637babe28d2c6441f83, 69e31b5ad78c4358b932f61f8dc1d662; no voters:
I20251027 14:52:02.983139 8544 raft_consensus.cc:2804] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:02.983158 8456 raft_consensus.cc:2468] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 69e31b5ad78c4358b932f61f8dc1d662 in term 1.
I20251027 14:52:02.983580 8544 raft_consensus.cc:697] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 LEADER]: Becoming Leader. State: Replica: 69e31b5ad78c4358b932f61f8dc1d662, State: Running, Role: LEADER
I20251027 14:52:02.984076 8544 consensus_queue.cc:237] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } }
I20251027 14:52:02.989739 7224 catalog_manager.cc:5649] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 reported cstate change: term changed from 0 to 1, leader changed from <none> to 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "69e31b5ad78c4358b932f61f8dc1d662" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } health_report { overall_health: UNKNOWN } } }
I20251027 14:52:03.007272 8532 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:03.002779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35044 (local address 127.0.104.2:46289)
1027 14:52:03.002920 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:03.002925 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:03.003013 (+ 88us) server_negotiation.cc:408] Connection header received
1027 14:52:03.003174 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:03.003178 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:03.003262 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:03.003375 (+ 113us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:03.003837 (+ 462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:03.004761 (+ 924us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:03.005896 (+ 1135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:03.006203 (+ 307us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:03.006343 (+ 140us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:03.006871 (+ 528us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:03.006980 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:03.007019 (+ 39us) server_negotiation.cc:300] Negotiation successful
1027 14:52:03.007094 (+ 75us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":43}
I20251027 14:52:03.015316 8456 raft_consensus.cc:1275] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Refusing update from remote peer 69e31b5ad78c4358b932f61f8dc1d662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:03.015311 8173 raft_consensus.cc:1275] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Refusing update from remote peer 69e31b5ad78c4358b932f61f8dc1d662: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:03.016237 8548 consensus_queue.cc:1048] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:03.016501 8544 consensus_queue.cc:1048] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [LEADER]: Connected to new peer: Peer: permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:03.047235 7225 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50204:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:03.047434 7225 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50204:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:03.053773 7225 catalog_manager.cc:5936] T 00000000000000000000000000000000 P ebe7c276923347a3ae5401422b9dab66: Sending DeleteTablet for 3 replicas of tablet bee08dbe6d3a41898ed48c841e88b015
I20251027 14:52:03.055373 8294 tablet_service.cc:1552] Processing DeleteTablet for tablet bee08dbe6d3a41898ed48c841e88b015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:35038
I20251027 14:52:03.055513 8436 tablet_service.cc:1552] Processing DeleteTablet for tablet bee08dbe6d3a41898ed48c841e88b015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:44188
I20251027 14:52:03.056193 8153 tablet_service.cc:1552] Processing DeleteTablet for tablet bee08dbe6d3a41898ed48c841e88b015 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:03 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:36084
I20251027 14:52:03.057937 8561 tablet_replica.cc:333] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: stopping tablet replica
I20251027 14:52:03.058437 8561 raft_consensus.cc:2243] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:03.059109 8561 raft_consensus.cc:2272] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:03.061144 8561 ts_tablet_manager.cc:1916] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:03.063422 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7193
I20251027 14:52:03.064493 7255 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:03.064891 8563 tablet_replica.cc:333] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: stopping tablet replica
I20251027 14:52:03.065325 8563 raft_consensus.cc:2243] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:03.066285 8562 tablet_replica.cc:333] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: stopping tablet replica
I20251027 14:52:03.067245 8562 raft_consensus.cc:2243] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:03.067658 8563 raft_consensus.cc:2272] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:03.068135 8562 raft_consensus.cc:2272] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:03.070680 8563 ts_tablet_manager.cc:1916] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:03.071004 8561 ts_tablet_manager.cc:1929] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:03.071167 8561 log.cc:1199] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/bee08dbe6d3a41898ed48c841e88b015
I20251027 14:52:03.072341 8561 ts_tablet_manager.cc:1950] T bee08dbe6d3a41898ed48c841e88b015 P 69e31b5ad78c4358b932f61f8dc1d662: Deleting consensus metadata
I20251027 14:52:03.073206 8562 ts_tablet_manager.cc:1916] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:03.078110 8562 ts_tablet_manager.cc:1929] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:03.078269 8562 log.cc:1199] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/bee08dbe6d3a41898ed48c841e88b015
I20251027 14:52:03.078469 8563 ts_tablet_manager.cc:1929] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:03.078634 8563 log.cc:1199] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/bee08dbe6d3a41898ed48c841e88b015
I20251027 14:52:03.078773 8562 ts_tablet_manager.cc:1950] T bee08dbe6d3a41898ed48c841e88b015 P af3b3966dd764458af1f02e364788f7a: Deleting consensus metadata
I20251027 14:52:03.079124 8563 ts_tablet_manager.cc:1950] T bee08dbe6d3a41898ed48c841e88b015 P 45af0ea8283a4637babe28d2c6441f83: Deleting consensus metadata
I20251027 14:52:03.280682 7209 catalog_manager.cc:4985] TS 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535): tablet bee08dbe6d3a41898ed48c841e88b015 (table test-table [id=4cc5544649fd4239a9421966b0b76e2d]) successfully deleted
I20251027 14:52:03.281121 7211 catalog_manager.cc:4985] TS 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289): tablet bee08dbe6d3a41898ed48c841e88b015 (table test-table [id=4cc5544649fd4239a9421966b0b76e2d]) successfully deleted
I20251027 14:52:03.281471 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7193
I20251027 14:52:03.359144 8570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:03.310398 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46746 (local address 127.0.104.60:43831)
1027 14:52:03.342285 (+ 31887us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:03.342294 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:03.343409 (+ 1115us) server_negotiation.cc:408] Connection header received
1027 14:52:03.343693 (+ 284us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:03.343700 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:03.343777 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:03.343854 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:03.346129 (+ 2275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:03.347445 (+ 1316us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:03.349110 (+ 1665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:03.349571 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:03.352748 (+ 3177us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:03.352797 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:03.352803 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:03.352847 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:03.355969 (+ 3122us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:03.356815 (+ 846us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:03.356822 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:03.356825 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:03.356928 (+ 103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:03.357286 (+ 358us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:03.357292 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:03.357296 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:03.357604 (+ 308us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:03.357814 (+ 210us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:03.358341 (+ 527us) server_negotiation.cc:300] Negotiation successful
1027 14:52:03.358777 (+ 436us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31713,"thread_start_us":31908,"threads_started":1}
I20251027 14:52:03.363277 8572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:03.343572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51386 (local address 127.0.104.61:37569)
1027 14:52:03.349633 (+ 6061us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:03.349637 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:03.349654 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:52:03.349701 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:03.349705 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:03.349774 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:03.349876 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:03.351774 (+ 1898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:03.352729 (+ 955us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:03.354230 (+ 1501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:03.354715 (+ 485us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:03.356902 (+ 2187us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:03.356946 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:03.356952 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:03.356990 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:03.360912 (+ 3922us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:03.361620 (+ 708us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:03.361624 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:03.361627 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:03.361695 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:03.362046 (+ 351us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:03.362051 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:03.362053 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:03.362330 (+ 277us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:03.362516 (+ 186us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:03.362819 (+ 303us) server_negotiation.cc:300] Negotiation successful
1027 14:52:03.362990 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":5970,"thread_start_us":5806,"threads_started":1}
I20251027 14:52:04.035737 8573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.035068 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:58033)
1027 14:52:04.035473 (+ 405us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.035571 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":302,"thread_start_us":201,"threads_started":1}
W20251027 14:52:04.036139 8360 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:44599 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.039311 8574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.038679 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:39985)
1027 14:52:04.039021 (+ 342us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.039107 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":259,"thread_start_us":177,"threads_started":1}
W20251027 14:52:04.039772 8219 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:44599 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.045523 8575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.044285 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:44297)
1027 14:52:04.045233 (+ 948us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.045361 (+ 128us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":844,"thread_start_us":762,"threads_started":1}
W20251027 14:52:04.045852 8502 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:44599 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.554791 8576 raft_consensus.cc:493] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ebe7c276923347a3ae5401422b9dab66)
I20251027 14:52:04.554992 8577 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Starting pre-election (detected failure of leader ebe7c276923347a3ae5401422b9dab66)
I20251027 14:52:04.555012 8576 raft_consensus.cc:515] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:04.555187 8577 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:04.555815 8576 leader_election.cc:290] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61:37569)
I20251027 14:52:04.555960 8577 leader_election.cc:290] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), d93ee0148b8e47c09454292e9e2db849 (127.0.104.60:43831)
I20251027 14:52:04.556828 8032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1681ae8013c442a9aaef00ecede5f9ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d93ee0148b8e47c09454292e9e2db849" is_pre_election: true
I20251027 14:52:04.556957 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.556002 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50614)
1027 14:52:04.556555 (+ 553us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.556696 (+ 141us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":438,"thread_start_us":310,"threads_started":1}
I20251027 14:52:04.557171 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.556075 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50620)
1027 14:52:04.556821 (+ 746us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.556945 (+ 124us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":644,"thread_start_us":528,"threads_started":1}
I20251027 14:52:04.557163 8032 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1681ae8013c442a9aaef00ecede5f9ef in term 1.
I20251027 14:52:04.557137 7844 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93ee0148b8e47c09454292e9e2db849" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1681ae8013c442a9aaef00ecede5f9ef" is_pre_election: true
I20251027 14:52:04.557448 7844 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93ee0148b8e47c09454292e9e2db849 in term 1.
W20251027 14:52:04.557431 8007 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.557631 7821 leader_election.cc:304] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1681ae8013c442a9aaef00ecede5f9ef, d93ee0148b8e47c09454292e9e2db849; no voters:
W20251027 14:52:04.557708 7819 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.557890 8577 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 14:52:04.558010 8577 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Starting leader election (detected failure of leader ebe7c276923347a3ae5401422b9dab66)
I20251027 14:52:04.558131 8577 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 1 FOLLOWER]: Advancing to term 2
I20251027 14:52:04.558844 8007 leader_election.cc:304] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1681ae8013c442a9aaef00ecede5f9ef, d93ee0148b8e47c09454292e9e2db849; no voters:
I20251027 14:52:04.559158 8576 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20251027 14:52:04.559319 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.558626 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50622)
1027 14:52:04.558759 (+ 133us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.558838 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20251027 14:52:04.559298 8576 raft_consensus.cc:493] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Starting leader election (detected failure of leader ebe7c276923347a3ae5401422b9dab66)
I20251027 14:52:04.559406 8576 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 1 FOLLOWER]: Advancing to term 2
W20251027 14:52:04.559490 8007 leader_election.cc:336] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.560550 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.560052 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50632)
1027 14:52:04.560298 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.560403 (+ 105us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":142}
W20251027 14:52:04.560742 7819 leader_election.cc:336] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 2 pre-election: RPC error from VoteRequest() call to peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.561280 8577 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:04.561786 8577 leader_election.cc:290] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 2 election: Requested vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), d93ee0148b8e47c09454292e9e2db849 (127.0.104.60:43831)
I20251027 14:52:04.562197 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.561870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50646)
1027 14:52:04.561988 (+ 118us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.562056 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
I20251027 14:52:04.562146 8576 raft_consensus.cc:515] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:04.562228 8032 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "1681ae8013c442a9aaef00ecede5f9ef" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "d93ee0148b8e47c09454292e9e2db849"
I20251027 14:52:04.562505 8032 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate 1681ae8013c442a9aaef00ecede5f9ef in current term 2: Already voted for candidate d93ee0148b8e47c09454292e9e2db849 in this term.
I20251027 14:52:04.562821 8576 leader_election.cc:290] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 2 election: Requested vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61:37569)
I20251027 14:52:04.563143 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.562833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50648)
1027 14:52:04.562953 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.563022 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
I20251027 14:52:04.563381 7844 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93ee0148b8e47c09454292e9e2db849" candidate_term: 2 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1681ae8013c442a9aaef00ecede5f9ef"
I20251027 14:52:04.563619 7844 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 2 FOLLOWER]: Leader election vote request: Denying vote to candidate d93ee0148b8e47c09454292e9e2db849 in current term 2: Already voted for candidate 1681ae8013c442a9aaef00ecede5f9ef in this term.
I20251027 14:52:04.563750 8579 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.563240 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50658)
1027 14:52:04.563562 (+ 322us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.563634 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":214}
I20251027 14:52:04.564293 8578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:04.563976 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50666)
1027 14:52:04.564107 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:04.564178 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251027 14:52:04.564282 7819 leader_election.cc:336] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.564399 7819 leader_election.cc:304] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 1681ae8013c442a9aaef00ecede5f9ef; no voters: d93ee0148b8e47c09454292e9e2db849, ebe7c276923347a3ae5401422b9dab66
I20251027 14:52:04.564656 8577 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
W20251027 14:52:04.564754 8007 leader_election.cc:336] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 2 election: RPC error from VoteRequest() call to peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:04.564878 8007 leader_election.cc:304] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 2 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d93ee0148b8e47c09454292e9e2db849; no voters: 1681ae8013c442a9aaef00ecede5f9ef, ebe7c276923347a3ae5401422b9dab66
I20251027 14:52:04.565119 8576 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Leader election lost for term 2. Reason: could not achieve majority
I20251027 14:52:05.037388 8582 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:05.036723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:33935)
1027 14:52:05.037135 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:05.037212 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":296,"thread_start_us":200,"threads_started":1}
I20251027 14:52:05.041110 8583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:05.040424 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:36887)
1027 14:52:05.040849 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:05.040928 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":324,"thread_start_us":225,"threads_started":1}
I20251027 14:52:05.047098 8584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:05.046454 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:53231)
1027 14:52:05.046838 (+ 384us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:05.046919 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":282,"thread_start_us":190,"threads_started":1}
I20251027 14:52:06.038940 8585 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.038197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:60431)
1027 14:52:06.038613 (+ 416us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.038710 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":214,"threads_started":1}
I20251027 14:52:06.042585 8586 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.041876 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:54029)
1027 14:52:06.042304 (+ 428us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.042403 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":224,"threads_started":1}
I20251027 14:52:06.048699 8587 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.047971 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:42533)
1027 14:52:06.048424 (+ 453us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.048519 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":189,"threads_started":1}
I20251027 14:52:06.556239 8588 raft_consensus.cc:493] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:06.556491 8588 raft_consensus.cc:515] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:06.557114 8588 leader_election.cc:290] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61:37569)
I20251027 14:52:06.558068 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.557295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50946)
1027 14:52:06.557758 (+ 463us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.557874 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":250,"threads_started":1}
I20251027 14:52:06.558223 7844 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93ee0148b8e47c09454292e9e2db849" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1681ae8013c442a9aaef00ecede5f9ef" is_pre_election: true
I20251027 14:52:06.558476 7844 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d93ee0148b8e47c09454292e9e2db849 in term 2.
I20251027 14:52:06.558920 8007 leader_election.cc:304] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1681ae8013c442a9aaef00ecede5f9ef, d93ee0148b8e47c09454292e9e2db849; no voters:
I20251027 14:52:06.559234 8588 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20251027 14:52:06.559351 8588 raft_consensus.cc:493] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:06.559484 8588 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 2 FOLLOWER]: Advancing to term 3
I20251027 14:52:06.559669 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.559299 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50958)
1027 14:52:06.559441 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.559523 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251027 14:52:06.559903 8007 leader_election.cc:336] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 3 pre-election: RPC error from VoteRequest() call to peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:06.561771 8588 raft_consensus.cc:515] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 3 FOLLOWER]: Starting leader election with config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:06.562283 8588 leader_election.cc:290] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 3 election: Requested vote from peers ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599), 1681ae8013c442a9aaef00ecede5f9ef (127.0.104.61:37569)
I20251027 14:52:06.562749 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.562387 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50970)
1027 14:52:06.562567 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.562630 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20251027 14:52:06.562947 7844 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d93ee0148b8e47c09454292e9e2db849" candidate_term: 3 candidate_status { last_received { term: 1 index: 12 } } ignore_live_leader: false dest_uuid: "1681ae8013c442a9aaef00ecede5f9ef"
I20251027 14:52:06.563138 7844 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 2 FOLLOWER]: Advancing to term 3
I20251027 14:52:06.563587 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.563304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50974)
1027 14:52:06.563418 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.563473 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251027 14:52:06.563759 8007 leader_election.cc:336] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 3 election: RPC error from VoteRequest() call to peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
I20251027 14:52:06.566174 7844 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d93ee0148b8e47c09454292e9e2db849 in term 3.
I20251027 14:52:06.566658 8007 leader_election.cc:304] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 1681ae8013c442a9aaef00ecede5f9ef, d93ee0148b8e47c09454292e9e2db849; no voters: ebe7c276923347a3ae5401422b9dab66
I20251027 14:52:06.566946 8588 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 14:52:06.567365 8588 raft_consensus.cc:697] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [term 3 LEADER]: Becoming Leader. State: Replica: d93ee0148b8e47c09454292e9e2db849, State: Running, Role: LEADER
I20251027 14:52:06.567747 8588 consensus_queue.cc:237] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 12, Committed index: 12, Last appended: 1.12, Last appended by leader: 12, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } }
I20251027 14:52:06.569669 8591 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93ee0148b8e47c09454292e9e2db849. Latest consensus state: current_term: 3 leader_uuid: "d93ee0148b8e47c09454292e9e2db849" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:06.569994 8591 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:06.570582 8593 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:06.574044 8593 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:06.574800 8593 catalog_manager.cc:1269] Loaded cluster ID: 1d1d091730ea451bb60f617614e64065
I20251027 14:52:06.574867 8593 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:06.575901 8593 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:06.576680 8593 catalog_manager.cc:6033] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Loaded TSK: 0
I20251027 14:52:06.577455 8593 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:52:06.621367 8021 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46746:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:52:06.625916 8021 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:06.630116 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.629730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:50990)
1027 14:52:06.629869 (+ 139us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.629973 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20251027 14:52:06.630457 7844 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [term 3 FOLLOWER]: Refusing update from remote peer d93ee0148b8e47c09454292e9e2db849: Log matching property violated. Preceding OpId in replica: term: 1 index: 12. Preceding OpId from leader: term: 3 index: 14. (index mismatch)
I20251027 14:52:06.631090 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.630778 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:51000)
1027 14:52:06.630891 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.630962 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20251027 14:52:06.631151 8588 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 13, Last known committed idx: 12, Time since last communication: 0.000s
W20251027 14:52:06.631456 8007 consensus_peers.cc:597] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 -> Peer ebe7c276923347a3ae5401422b9dab66 (127.0.104.62:44599): Couldn't send request to peer ebe7c276923347a3ae5401422b9dab66. Status: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20251027 14:52:06.633736 8595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: New leader d93ee0148b8e47c09454292e9e2db849. Latest consensus state: current_term: 3 leader_uuid: "d93ee0148b8e47c09454292e9e2db849" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:06.633944 8595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:52:06.635864 8588 sys_catalog.cc:455] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d93ee0148b8e47c09454292e9e2db849" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:06.636135 8588 sys_catalog.cc:458] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:06.636350 8595 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d93ee0148b8e47c09454292e9e2db849" committed_config { opid_index: 8 OBSOLETE_local: true peers { permanent_uuid: "ebe7c276923347a3ae5401422b9dab66" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44599 } } peers { permanent_uuid: "1681ae8013c442a9aaef00ecede5f9ef" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 37569 } attrs { promote: false } } peers { permanent_uuid: "d93ee0148b8e47c09454292e9e2db849" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43831 } attrs { promote: false } } }
I20251027 14:52:06.636623 8595 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1681ae8013c442a9aaef00ecede5f9ef [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:52:06.650022 8589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.644109 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:35535 (local address 127.0.0.1:36088)
1027 14:52:06.644254 (+ 145us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.644277 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:06.644404 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:06.644841 (+ 437us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:06.644846 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:06.644871 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:06.645663 (+ 792us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.645679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.647415 (+ 1736us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.647422 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:06.649294 (+ 1872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.649310 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.649474 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.649503 (+ 29us) client_negotiation.cc:769] Sending connection context
1027 14:52:06.649569 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:52:06.649708 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":40}
I20251027 14:52:06.651190 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.644121 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36088 (local address 127.0.104.1:35535)
1027 14:52:06.644538 (+ 417us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.644544 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.644562 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:52:06.644628 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.644634 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.644709 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.644826 (+ 117us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:06.645819 (+ 993us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.647220 (+ 1401us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.649471 (+ 2251us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.650572 (+ 1101us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.650628 (+ 56us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.650711 (+ 83us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.650840 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":320,"thread_start_us":216,"threads_started":1}
I20251027 14:52:06.652029 8153 tablet_service.cc:1505] Processing CreateTablet for tablet 23c7d458e7c94ba39ae3ff51151b878a (DEFAULT_TABLE table=test-table [id=f0da31ca5d9944909db9bb86f61a4b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:06.652554 8153 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c7d458e7c94ba39ae3ff51151b878a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:06.660539 8606 tablet_bootstrap.cc:492] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Bootstrap starting.
I20251027 14:52:06.662230 8604 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.645695 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:46289 (local address 127.0.0.1:35048)
1027 14:52:06.646241 (+ 546us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.646257 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:06.646363 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:06.657686 (+ 11323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:06.657691 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:06.657718 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:06.658318 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.658330 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.660077 (+ 1747us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.660084 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:06.661497 (+ 1413us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.661508 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.661684 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.661711 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:06.661780 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:52:06.661911 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":554,"threads_started":1}
I20251027 14:52:06.662331 8606 tablet_bootstrap.cc:654] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:06.663496 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.644932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:45563 (local address 127.0.0.1:44192)
1027 14:52:06.645459 (+ 527us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.645475 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:06.645589 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:06.658758 (+ 13169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:06.658763 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:06.658786 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:06.659415 (+ 629us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.659429 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.661099 (+ 1670us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.661105 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:06.662786 (+ 1681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.662800 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.662972 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.662996 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:06.663060 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:52:06.663171 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":509,"threads_started":1}
I20251027 14:52:06.663786 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.646112 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35048 (local address 127.0.104.2:46289)
1027 14:52:06.657332 (+ 11220us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.657339 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.657362 (+ 23us) server_negotiation.cc:408] Connection header received
1027 14:52:06.657440 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.657446 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.657522 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.657647 (+ 125us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:06.658474 (+ 827us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.659895 (+ 1421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.661658 (+ 1763us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.663246 (+ 1588us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.663295 (+ 49us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.663382 (+ 87us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.663510 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11099,"thread_start_us":11631,"threads_started":1}
I20251027 14:52:06.664567 8294 tablet_service.cc:1505] Processing CreateTablet for tablet 23c7d458e7c94ba39ae3ff51151b878a (DEFAULT_TABLE table=test-table [id=f0da31ca5d9944909db9bb86f61a4b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:06.664927 8294 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c7d458e7c94ba39ae3ff51151b878a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:06.664988 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.645288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44192 (local address 127.0.104.3:45563)
1027 14:52:06.657584 (+ 12296us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.657589 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.657605 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:52:06.657671 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.657676 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.657749 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.657864 (+ 115us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:06.659580 (+ 1716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.660853 (+ 1273us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.663435 (+ 2582us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.664456 (+ 1021us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.664508 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.664588 (+ 80us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.664711 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12201,"thread_start_us":12678,"threads_started":1}
I20251027 14:52:06.665392 8606 tablet_bootstrap.cc:492] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: No bootstrap required, opened a new log
I20251027 14:52:06.665518 8606 ts_tablet_manager.cc:1403] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20251027 14:52:06.666152 8606 raft_consensus.cc:359] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.666291 8606 raft_consensus.cc:385] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:06.666373 8606 raft_consensus.cc:740] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 45af0ea8283a4637babe28d2c6441f83, State: Initialized, Role: FOLLOWER
I20251027 14:52:06.666566 8606 consensus_queue.cc:260] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.667207 8606 ts_tablet_manager.cc:1434] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 14:52:06.668491 8436 tablet_service.cc:1505] Processing CreateTablet for tablet 23c7d458e7c94ba39ae3ff51151b878a (DEFAULT_TABLE table=test-table [id=f0da31ca5d9944909db9bb86f61a4b8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:06.668915 8436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 23c7d458e7c94ba39ae3ff51151b878a. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:06.669215 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.667824 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:52623)
1027 14:52:06.668818 (+ 994us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.668961 (+ 143us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":878,"thread_start_us":779,"threads_started":1}
I20251027 14:52:06.670819 8218 heartbeater.cc:499] Master 127.0.104.60:43831 was elected leader, sending a full tablet report...
I20251027 14:52:06.671506 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.670310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:54747)
1027 14:52:06.670457 (+ 147us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.670534 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20251027 14:52:06.675040 8611 tablet_bootstrap.cc:492] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Bootstrap starting.
I20251027 14:52:06.675318 8610 tablet_bootstrap.cc:492] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Bootstrap starting.
I20251027 14:52:06.677089 8611 tablet_bootstrap.cc:654] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:06.677089 8610 tablet_bootstrap.cc:654] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:06.679395 8611 tablet_bootstrap.cc:492] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: No bootstrap required, opened a new log
I20251027 14:52:06.679397 8610 tablet_bootstrap.cc:492] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: No bootstrap required, opened a new log
I20251027 14:52:06.679543 8610 ts_tablet_manager.cc:1403] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:52:06.679543 8611 ts_tablet_manager.cc:1403] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20251027 14:52:06.680323 8611 raft_consensus.cc:359] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.680330 8610 raft_consensus.cc:359] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.680557 8610 raft_consensus.cc:385] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:06.680562 8611 raft_consensus.cc:385] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:06.680653 8610 raft_consensus.cc:740] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 69e31b5ad78c4358b932f61f8dc1d662, State: Initialized, Role: FOLLOWER
I20251027 14:52:06.680653 8611 raft_consensus.cc:740] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: af3b3966dd764458af1f02e364788f7a, State: Initialized, Role: FOLLOWER
I20251027 14:52:06.680876 8611 consensus_queue.cc:260] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.680887 8610 consensus_queue.cc:260] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.681639 8611 ts_tablet_manager.cc:1434] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251027 14:52:06.682646 8610 ts_tablet_manager.cc:1434] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:06.682747 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.681920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:59783)
1027 14:52:06.682367 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.682471 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":231,"threads_started":1}
I20251027 14:52:06.684156 8501 heartbeater.cc:499] Master 127.0.104.60:43831 was elected leader, sending a full tablet report...
I20251027 14:52:06.684178 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.683715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:34645)
1027 14:52:06.683920 (+ 205us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.684007 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":110}
I20251027 14:52:06.685171 8359 heartbeater.cc:499] Master 127.0.104.60:43831 was elected leader, sending a full tablet report...
I20251027 14:52:06.685385 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.683715 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:47277)
1027 14:52:06.684896 (+ 1181us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.685050 (+ 154us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":1045,"thread_start_us":944,"threads_started":1}
I20251027 14:52:06.687592 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.687209 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:48277)
1027 14:52:06.687347 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.687425 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20251027 14:52:06.816275 8608 raft_consensus.cc:493] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:06.816542 8608 raft_consensus.cc:515] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.818115 8608 leader_election.cc:290] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563), 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289)
W20251027 14:52:06.821709 8221 tablet.cc:2378] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:06.823130 8609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.818249 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:45563 (local address 127.0.104.1:45545)
1027 14:52:06.818496 (+ 247us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.818515 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:06.818685 (+ 170us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:06.819114 (+ 429us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:06.819120 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:06.819147 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:06.819634 (+ 487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.819643 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.820896 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.820901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:06.822434 (+ 1533us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.822448 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.822610 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.822637 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:06.822708 (+ 71us) client_negotiation.cc:240] Negotiation successful
1027 14:52:06.822840 (+ 132us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":134}
I20251027 14:52:06.823487 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.818441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:46289 (local address 127.0.104.1:54265)
1027 14:52:06.819086 (+ 645us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.819100 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:06.819232 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:06.819541 (+ 309us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:06.819545 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:06.819568 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:06.820114 (+ 546us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.820123 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.821397 (+ 1274us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.821401 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:06.822785 (+ 1384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:06.822797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.822975 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.823001 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:52:06.823067 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:52:06.823183 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"thread_start_us":363,"threads_started":1}
I20251027 14:52:06.824035 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.818366 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:45545 (local address 127.0.104.3:45563)
1027 14:52:06.818524 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.818530 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.818871 (+ 341us) server_negotiation.cc:408] Connection header received
1027 14:52:06.818935 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.818939 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.818995 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.819071 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:06.819785 (+ 714us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.820720 (+ 935us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.822593 (+ 1873us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.823636 (+ 1043us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.823687 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.823765 (+ 78us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.823873 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:52:06.824352 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.818683 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:54265 (local address 127.0.104.2:46289)
1027 14:52:06.818839 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.818844 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.819194 (+ 350us) server_negotiation.cc:408] Connection header received
1027 14:52:06.819370 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.819375 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.819432 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.819503 (+ 71us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:06.820249 (+ 746us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.821204 (+ 955us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.822930 (+ 1726us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.823945 (+ 1015us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.823983 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.824045 (+ 62us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.824161 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:52:06.824599 8456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23c7d458e7c94ba39ae3ff51151b878a" candidate_uuid: "45af0ea8283a4637babe28d2c6441f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b3966dd764458af1f02e364788f7a" is_pre_election: true
I20251027 14:52:06.824790 8456 raft_consensus.cc:2468] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45af0ea8283a4637babe28d2c6441f83 in term 0.
I20251027 14:52:06.825176 8314 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23c7d458e7c94ba39ae3ff51151b878a" candidate_uuid: "45af0ea8283a4637babe28d2c6441f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e31b5ad78c4358b932f61f8dc1d662" is_pre_election: true
I20251027 14:52:06.825340 8109 leader_election.cc:304] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45af0ea8283a4637babe28d2c6441f83, af3b3966dd764458af1f02e364788f7a; no voters:
I20251027 14:52:06.825518 8314 raft_consensus.cc:2468] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 45af0ea8283a4637babe28d2c6441f83 in term 0.
I20251027 14:52:06.825719 8608 raft_consensus.cc:2804] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:06.825830 8608 raft_consensus.cc:493] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:06.825954 8608 raft_consensus.cc:3060] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:06.828249 8608 raft_consensus.cc:515] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.828842 8608 leader_election.cc:290] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [CANDIDATE]: Term 1 election: Requested vote from peers af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563), 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289)
I20251027 14:52:06.829304 8456 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23c7d458e7c94ba39ae3ff51151b878a" candidate_uuid: "45af0ea8283a4637babe28d2c6441f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "af3b3966dd764458af1f02e364788f7a"
I20251027 14:52:06.829360 8314 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "23c7d458e7c94ba39ae3ff51151b878a" candidate_uuid: "45af0ea8283a4637babe28d2c6441f83" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "69e31b5ad78c4358b932f61f8dc1d662"
I20251027 14:52:06.829463 8456 raft_consensus.cc:3060] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:06.829538 8314 raft_consensus.cc:3060] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:06.831517 8456 raft_consensus.cc:2468] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45af0ea8283a4637babe28d2c6441f83 in term 1.
I20251027 14:52:06.831750 8314 raft_consensus.cc:2468] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 45af0ea8283a4637babe28d2c6441f83 in term 1.
I20251027 14:52:06.831985 8109 leader_election.cc:304] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 45af0ea8283a4637babe28d2c6441f83, af3b3966dd764458af1f02e364788f7a; no voters:
I20251027 14:52:06.832285 8608 raft_consensus.cc:2804] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:06.832753 8608 raft_consensus.cc:697] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 1 LEADER]: Becoming Leader. State: Replica: 45af0ea8283a4637babe28d2c6441f83, State: Running, Role: LEADER
I20251027 14:52:06.833271 8608 consensus_queue.cc:237] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } }
I20251027 14:52:06.833832 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.833404 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:33573)
1027 14:52:06.833561 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.833662 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251027 14:52:06.837206 8022 catalog_manager.cc:5649] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 reported cstate change: term changed from 0 to 1, leader changed from <none> to 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "45af0ea8283a4637babe28d2c6441f83" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "45af0ea8283a4637babe28d2c6441f83" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 } health_report { overall_health: UNKNOWN } } }
I20251027 14:52:06.854089 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.849409 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36092 (local address 127.0.104.1:35535)
1027 14:52:06.849547 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.849551 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.849590 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:52:06.849785 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.849790 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.849865 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.850019 (+ 154us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:06.850561 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.851465 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.852643 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.852957 (+ 314us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.853024 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:06.853562 (+ 538us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:06.853665 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.853795 (+ 130us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.853871 (+ 76us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:52:06.858103 8456 raft_consensus.cc:1275] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Refusing update from remote peer 45af0ea8283a4637babe28d2c6441f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:06.858417 8314 raft_consensus.cc:1275] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Refusing update from remote peer 45af0ea8283a4637babe28d2c6441f83: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:06.858747 8608 consensus_queue.cc:1048] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "af3b3966dd764458af1f02e364788f7a" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45563 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:06.859113 8619 consensus_queue.cc:1048] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [LEADER]: Connected to new peer: Peer: permanent_uuid: "69e31b5ad78c4358b932f61f8dc1d662" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 46289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:06.862277 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.861576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:50639)
1027 14:52:06.861959 (+ 383us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.862046 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":280}
I20251027 14:52:06.862663 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.862208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:55489)
1027 14:52:06.862425 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.862503 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":127}
I20251027 14:52:06.866434 8623 mvcc.cc:204] Tried to move back new op lower bound from 7215418273204162560 to 7215418273113600000. Current Snapshot: MvccSnapshot[applied={T|T < 7215418273204162560}]
I20251027 14:52:06.867205 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.866720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:54875)
1027 14:52:06.866927 (+ 207us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.867017 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":117}
I20251027 14:52:06.867565 8616 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.867090 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.3:47465)
1027 14:52:06.867363 (+ 273us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.867444 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":191}
I20251027 14:52:06.880440 8617 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.879904 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.2:57509)
1027 14:52:06.880125 (+ 221us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.880232 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
I20251027 14:52:06.880863 8618 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.880316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.104.1:58139)
1027 14:52:06.880622 (+ 306us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:06.880718 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":209}
I20251027 14:52:06.887380 8022 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:06.887576 8022 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46746:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:06.893096 8022 catalog_manager.cc:5936] T 00000000000000000000000000000000 P d93ee0148b8e47c09454292e9e2db849: Sending DeleteTablet for 3 replicas of tablet 23c7d458e7c94ba39ae3ff51151b878a
I20251027 14:52:06.894078 8153 tablet_service.cc:1552] Processing DeleteTablet for tablet 23c7d458e7c94ba39ae3ff51151b878a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:06 UTC) from {username='oryx', principal='oryx/127.0.104.60@KRBTEST.COM'} at 127.0.0.1:36088
I20251027 14:52:06.894927 8436 tablet_service.cc:1552] Processing DeleteTablet for tablet 23c7d458e7c94ba39ae3ff51151b878a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:06 UTC) from {username='oryx', principal='oryx/127.0.104.60@KRBTEST.COM'} at 127.0.0.1:44192
I20251027 14:52:06.894956 8294 tablet_service.cc:1552] Processing DeleteTablet for tablet 23c7d458e7c94ba39ae3ff51151b878a with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:06 UTC) from {username='oryx', principal='oryx/127.0.104.60@KRBTEST.COM'} at 127.0.0.1:35048
I20251027 14:52:06.896263 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 8090
I20251027 14:52:06.908579 8632 tablet_replica.cc:333] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: stopping tablet replica
I20251027 14:52:06.908970 8632 raft_consensus.cc:2243] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:06.909778 8632 raft_consensus.cc:2272] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:06.911266 8632 ts_tablet_manager.cc:1916] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:06.912845 8634 tablet_replica.cc:333] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: stopping tablet replica
I20251027 14:52:06.913419 8633 tablet_replica.cc:333] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: stopping tablet replica
I20251027 14:52:06.913633 8633 raft_consensus.cc:2243] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:06.913908 8633 raft_consensus.cc:2272] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:06.914801 8633 ts_tablet_manager.cc:1916] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:06.921480 8632 ts_tablet_manager.cc:1929] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:06.921778 8634 raft_consensus.cc:2243] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:06.921882 8633 ts_tablet_manager.cc:1929] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:06.922060 8633 log.cc:1199] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/23c7d458e7c94ba39ae3ff51151b878a
I20251027 14:52:06.922113 8634 raft_consensus.cc:2272] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:06.922552 8633 ts_tablet_manager.cc:1950] T 23c7d458e7c94ba39ae3ff51151b878a P af3b3966dd764458af1f02e364788f7a: Deleting consensus metadata
I20251027 14:52:06.922675 8632 log.cc:1199] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/23c7d458e7c94ba39ae3ff51151b878a
I20251027 14:52:06.923168 8632 ts_tablet_manager.cc:1950] T 23c7d458e7c94ba39ae3ff51151b878a P 45af0ea8283a4637babe28d2c6441f83: Deleting consensus metadata
I20251027 14:52:06.923732 8009 catalog_manager.cc:4985] TS af3b3966dd764458af1f02e364788f7a (127.0.104.3:45563): tablet 23c7d458e7c94ba39ae3ff51151b878a (table test-table [id=f0da31ca5d9944909db9bb86f61a4b8b]) successfully deleted
I20251027 14:52:06.924573 8006 catalog_manager.cc:4985] TS 45af0ea8283a4637babe28d2c6441f83 (127.0.104.1:35535): tablet 23c7d458e7c94ba39ae3ff51151b878a (table test-table [id=f0da31ca5d9944909db9bb86f61a4b8b]) successfully deleted
I20251027 14:52:06.925060 8634 ts_tablet_manager.cc:1916] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:06.927193 8601 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:06.898472 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36098 (local address 127.0.104.1:35535)
1027 14:52:06.898643 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:06.898650 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:06.904868 (+ 6218us) server_negotiation.cc:408] Connection header received
1027 14:52:06.905083 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:06.905090 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:06.905204 (+ 114us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:06.905315 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:06.907743 (+ 2428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.909409 (+ 1666us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:06.911184 (+ 1775us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:06.911729 (+ 545us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:06.919112 (+ 7383us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:06.919180 (+ 68us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:06.919201 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:06.919251 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:06.923927 (+ 4676us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:06.925315 (+ 1388us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:06.925323 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:06.925326 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:06.925411 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:06.925727 (+ 316us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:06.925732 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:06.925734 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:06.926033 (+ 299us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:06.926329 (+ 296us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:06.926601 (+ 272us) server_negotiation.cc:300] Negotiation successful
1027 14:52:06.926852 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251027 14:52:06.927793 8213 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:06.931730 8634 ts_tablet_manager.cc:1929] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:06.931859 8634 log.cc:1199] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipalMultipleMaster.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/23c7d458e7c94ba39ae3ff51151b878a
I20251027 14:52:06.932199 8634 ts_tablet_manager.cc:1950] T 23c7d458e7c94ba39ae3ff51151b878a P 69e31b5ad78c4358b932f61f8dc1d662: Deleting consensus metadata
I20251027 14:52:06.933175 8008 catalog_manager.cc:4985] TS 69e31b5ad78c4358b932f61f8dc1d662 (127.0.104.2:46289): tablet 23c7d458e7c94ba39ae3ff51151b878a (table test-table [id=f0da31ca5d9944909db9bb86f61a4b8b]) successfully deleted
I20251027 14:52:07.039479 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 8090
I20251027 14:52:07.054986 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 8231
I20251027 14:52:07.066689 8605 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:07.055726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35056 (local address 127.0.104.2:46289)
1027 14:52:07.055871 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:07.055875 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:07.055892 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:52:07.056012 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:07.056017 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:07.056079 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:07.056171 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:07.057613 (+ 1442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:07.058485 (+ 872us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:07.059496 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:07.059832 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:07.061666 (+ 1834us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:07.061703 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:07.061720 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:07.061748 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:07.064489 (+ 2741us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:07.065105 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:07.065111 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:07.065114 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:07.065206 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:07.065431 (+ 225us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:07.065433 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:07.065435 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:07.065720 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:07.065934 (+ 214us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:07.066232 (+ 298us) server_negotiation.cc:300] Negotiation successful
1027 14:52:07.066423 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:52:07.067390 8354 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:07.128458 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:07.127953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:51002)
1027 14:52:07.128125 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:07.128235 (+ 110us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
I20251027 14:52:07.129539 8602 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:07.129127 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:44599 (local address 127.0.0.1:51012)
1027 14:52:07.129279 (+ 152us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:07.129369 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:44599: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20251027 14:52:07.183101 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 8231
I20251027 14:52:07.201162 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 8372
I20251027 14:52:07.212663 8603 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:07.201845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44204 (local address 127.0.104.3:45563)
1027 14:52:07.202002 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:07.202006 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:07.202024 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:52:07.202171 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:07.202176 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:07.202239 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:07.202326 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:07.203715 (+ 1389us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:07.204672 (+ 957us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:07.205715 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:07.206077 (+ 362us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:07.207826 (+ 1749us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:07.207864 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:07.207880 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:07.207917 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:07.210597 (+ 2680us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:07.211187 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:07.211196 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:07.211198 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:07.211276 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:07.211496 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:07.211499 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:07.211501 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:07.211792 (+ 291us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:07.212037 (+ 245us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:07.212201 (+ 164us) server_negotiation.cc:300] Negotiation successful
1027 14:52:07.212371 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251027 14:52:07.213305 8496 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:07.307884 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 8372
I20251027 14:52:07.327456 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7695
I20251027 14:52:07.328723 7864 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:07.444890 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7695
I20251027 14:52:07.461781 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 7800
I20251027 14:52:07.462816 8052 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:07.583055 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 7800
2025-10-27T14:52:07Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipalMultipleMaster (14324 ms)
[ RUN ] SecurityITest.TestNonDefaultPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:52:07 dist-test-slave-hxxt krb5kdc[8643](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:52:07 dist-test-slave-hxxt krb5kdc[8643](info): set up 2 sockets
Oct 27 14:52:07 dist-test-slave-hxxt krb5kdc[8643](info): commencing operation
krb5kdc: starting...
W20251027 14:52:09.453061 416 mini_kdc.cc:121] Time spent starting KDC: real 1.829s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:52:09 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:52:09Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:09Z Disabled control of system clock
WARNING: no policy specified for oryx/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:52:09.707254 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34959
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34959
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--txn_manager_enabled=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:10.030689 8659 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:10.031028 8659 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:10.031158 8659 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:10.039399 8659 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:10.039508 8659 flags.cc:425] Enabled experimental flag: --txn_manager_enabled=true
W20251027 14:52:10.039587 8659 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:10.039669 8659 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:10.039748 8659 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:10.039806 8659 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:10.050877 8659 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34959
--txn_manager_enabled=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34959
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:10.052515 8659 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:10.053869 8659 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:10.062120 8664 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:10.062009 8665 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:10.062151 8667 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:10.063057 8659 server_base.cc:1047] running on GCE node
I20251027 14:52:10.063860 8659 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:10.065356 8659 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:10.066629 8659 hybrid_clock.cc:648] HybridClock initialized: now 1761576730066536 us; error 37 us; skew 500 ppm
I20251027 14:52:10.067005 8659 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:10 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576730, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:10.070117 8659 init.cc:377] Logged in from keytab as oryx/127.0.104.62@KRBTEST.COM (short username oryx)
I20251027 14:52:10.071774 8659 webserver.cc:492] Webserver started at http://127.0.104.62:33871/ using document root <none> and password file <none>
I20251027 14:52:10.072544 8659 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:10.072667 8659 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:10.073006 8659 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:10.075595 8659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "09e138be90f842f098cec4a16f28e5ed"
format_stamp: "Formatted at 2025-10-27 14:52:10 on dist-test-slave-hxxt"
server_key: "75bee1bca1cd2a3801a86bc792dc6414"
server_key_iv: "308f1e98222e3228a5ed95973ec6400d"
server_key_version: "encryptionkey@0"
I20251027 14:52:10.076439 8659 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "09e138be90f842f098cec4a16f28e5ed"
format_stamp: "Formatted at 2025-10-27 14:52:10 on dist-test-slave-hxxt"
server_key: "75bee1bca1cd2a3801a86bc792dc6414"
server_key_iv: "308f1e98222e3228a5ed95973ec6400d"
server_key_version: "encryptionkey@0"
I20251027 14:52:10.081964 8659 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20251027 14:52:10.085517 8674 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:10.086931 8659 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:52:10.087105 8659 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "09e138be90f842f098cec4a16f28e5ed"
format_stamp: "Formatted at 2025-10-27 14:52:10 on dist-test-slave-hxxt"
server_key: "75bee1bca1cd2a3801a86bc792dc6414"
server_key_iv: "308f1e98222e3228a5ed95973ec6400d"
server_key_version: "encryptionkey@0"
I20251027 14:52:10.087301 8659 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:10.099028 8659 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:10.103407 8659 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:10.103756 8659 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:10.126683 8659 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34959
I20251027 14:52:10.126746 8735 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34959 every 8 connection(s)
I20251027 14:52:10.128196 8659 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:52:10.132436 8736 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:10.137902 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 8659
I20251027 14:52:10.138100 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:52:10.138635 416 external_mini_cluster.cc:1468] Setting key 5f94cb968be700122b8241edb8f64e3e
I20251027 14:52:10.143167 8736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Bootstrap starting.
I20251027 14:52:10.146210 8736 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:10.147259 8736 log.cc:826] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:10.150415 8736 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: No bootstrap required, opened a new log
Oct 27 14:52:10 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576729, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:10.157416 8736 raft_consensus.cc:359] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e138be90f842f098cec4a16f28e5ed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34959 } }
I20251027 14:52:10.157876 8736 raft_consensus.cc:385] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:10.157999 8736 raft_consensus.cc:740] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 09e138be90f842f098cec4a16f28e5ed, State: Initialized, Role: FOLLOWER
I20251027 14:52:10.158705 8736 consensus_queue.cc:260] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e138be90f842f098cec4a16f28e5ed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34959 } }
I20251027 14:52:10.158957 8736 raft_consensus.cc:399] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:52:10.159106 8736 raft_consensus.cc:493] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:52:10.159279 8736 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:10.159868 8739 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:10.140626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40746 (local address 127.0.104.62:34959)
1027 14:52:10.141457 (+ 831us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:10.141470 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:10.141513 (+ 43us) server_negotiation.cc:408] Connection header received
1027 14:52:10.142740 (+ 1227us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:10.142774 (+ 34us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:10.143197 (+ 423us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:10.143617 (+ 420us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:10.145244 (+ 1627us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.146640 (+ 1396us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:10.147832 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.148361 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:10.151653 (+ 3292us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:10.151730 (+ 77us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:10.151751 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:10.151800 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:10.155658 (+ 3858us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:10.156324 (+ 666us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:10.156333 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:10.156342 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:10.156492 (+ 150us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:10.156776 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:10.156781 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:10.156784 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:10.157370 (+ 586us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:10.157663 (+ 293us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:10.157984 (+ 321us) server_negotiation.cc:300] Negotiation successful
1027 14:52:10.158465 (+ 481us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":625,"thread_start_us":420,"threads_started":1}
I20251027 14:52:10.161917 8736 raft_consensus.cc:515] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e138be90f842f098cec4a16f28e5ed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34959 } }
I20251027 14:52:10.162456 8736 leader_election.cc:304] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 09e138be90f842f098cec4a16f28e5ed; no voters:
I20251027 14:52:10.163069 8736 leader_election.cc:290] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:52:10.163169 8741 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:10.163760 8741 raft_consensus.cc:697] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [term 1 LEADER]: Becoming Leader. State: Replica: 09e138be90f842f098cec4a16f28e5ed, State: Running, Role: LEADER
I20251027 14:52:10.164304 8741 consensus_queue.cc:237] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e138be90f842f098cec4a16f28e5ed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34959 } }
I20251027 14:52:10.165073 8736 sys_catalog.cc:565] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:52:10.167094 8743 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 09e138be90f842f098cec4a16f28e5ed. Latest consensus state: current_term: 1 leader_uuid: "09e138be90f842f098cec4a16f28e5ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e138be90f842f098cec4a16f28e5ed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34959 } } }
I20251027 14:52:10.167104 8742 sys_catalog.cc:455] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "09e138be90f842f098cec4a16f28e5ed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "09e138be90f842f098cec4a16f28e5ed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34959 } } }
I20251027 14:52:10.167495 8743 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:10.167490 8742 sys_catalog.cc:458] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:10.170468 8746 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:10.176913 8746 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:10.185690 8746 catalog_manager.cc:1357] Generated new cluster ID: 1c38644b8bdd4077896833e8b9903d1a
I20251027 14:52:10.185812 8746 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:10.196978 8746 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:52:10.198756 8746 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:10.210788 8746 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Generated new TSK 0
I20251027 14:52:10.211671 8746 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for oryx/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:52:10.309944 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34959
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:10.639679 8764 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:10.640007 8764 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:10.640151 8764 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:10.648809 8764 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:10.648928 8764 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:10.649081 8764 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:52:10.649183 8764 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:10.659570 8764 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:10.661484 8764 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:10.662825 8764 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:10.672212 8770 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:10.672214 8769 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:10.673483 8772 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:10.673977 8764 server_base.cc:1047] running on GCE node
I20251027 14:52:10.674530 8764 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:10.675454 8764 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:10.676651 8764 hybrid_clock.cc:648] HybridClock initialized: now 1761576730676595 us; error 41 us; skew 500 ppm
I20251027 14:52:10.676929 8764 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:10 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576730, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:10.679992 8764 init.cc:377] Logged in from keytab as oryx/127.0.104.1@KRBTEST.COM (short username oryx)
I20251027 14:52:10.681592 8764 webserver.cc:492] Webserver started at http://127.0.104.1:44355/ using document root <none> and password file <none>
I20251027 14:52:10.682320 8764 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:10.682426 8764 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:10.682792 8764 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:10.685433 8764 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "1fa892c685384a6195fe18e0ad41f885"
format_stamp: "Formatted at 2025-10-27 14:52:10 on dist-test-slave-hxxt"
server_key: "dbcac0c244f3d95154e84b3d8eefb1c1"
server_key_iv: "44c36727a1c3f55feb860a3d84a76f4e"
server_key_version: "encryptionkey@0"
I20251027 14:52:10.686215 8764 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "1fa892c685384a6195fe18e0ad41f885"
format_stamp: "Formatted at 2025-10-27 14:52:10 on dist-test-slave-hxxt"
server_key: "dbcac0c244f3d95154e84b3d8eefb1c1"
server_key_iv: "44c36727a1c3f55feb860a3d84a76f4e"
server_key_version: "encryptionkey@0"
I20251027 14:52:10.691640 8764 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:52:10.695037 8779 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:10.696585 8764 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:10.696745 8764 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "1fa892c685384a6195fe18e0ad41f885"
format_stamp: "Formatted at 2025-10-27 14:52:10 on dist-test-slave-hxxt"
server_key: "dbcac0c244f3d95154e84b3d8eefb1c1"
server_key_iv: "44c36727a1c3f55feb860a3d84a76f4e"
server_key_version: "encryptionkey@0"
I20251027 14:52:10.696928 8764 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:10.714021 8764 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:10.718076 8764 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:10.718346 8764 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:10.721158 8764 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:10.721285 8764 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:10.721400 8764 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:10.721480 8764 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:52:10 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576730, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:10.742301 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:10.722159 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:59119 (local address 127.0.104.62:34959)
1027 14:52:10.722556 (+ 397us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:10.722560 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:10.723811 (+ 1251us) server_negotiation.cc:408] Connection header received
1027 14:52:10.725539 (+ 1728us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:10.725601 (+ 62us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:10.725685 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:10.725788 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:10.727952 (+ 2164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.729203 (+ 1251us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:10.730937 (+ 1734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.731429 (+ 492us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:10.735795 (+ 4366us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:10.735833 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:10.735837 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:10.735871 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:10.739087 (+ 3216us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:10.739857 (+ 770us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:10.739863 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:10.739866 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:10.739975 (+ 109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:10.740522 (+ 547us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:10.740529 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:10.740532 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:10.740802 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:10.740939 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:10.741708 (+ 769us) server_negotiation.cc:300] Negotiation successful
1027 14:52:10.741951 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":291,"thread_start_us":212,"threads_started":1}
I20251027 14:52:10.743669 8789 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:10.722775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.104.1:59119)
1027 14:52:10.723642 (+ 867us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:10.723681 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:10.725239 (+ 1558us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:10.726071 (+ 832us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:10.726080 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:10.726726 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:10.727733 (+ 1007us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:10.727768 (+ 35us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.729365 (+ 1597us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:10.729369 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:10.730732 (+ 1363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:10.730744 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.731098 (+ 354us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:10.732004 (+ 906us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:10.732042 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:10.735534 (+ 3492us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:10.739268 (+ 3734us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:10.739278 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:10.739295 (+ 17us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:10.739726 (+ 431us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:10.740125 (+ 399us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:10.740132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:10.740138 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:10.740346 (+ 208us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:10.740937 (+ 591us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:10.740947 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:10.741281 (+ 334us) client_negotiation.cc:769] Sending connection context
1027 14:52:10.741579 (+ 298us) client_negotiation.cc:240] Negotiation successful
1027 14:52:10.742080 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":615,"thread_start_us":424,"threads_started":1}
I20251027 14:52:10.764317 8853 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:10.750446 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.104.1:46237)
1027 14:52:10.751258 (+ 812us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:10.751274 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:10.751430 (+ 156us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:10.751855 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:10.751861 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:10.752694 (+ 833us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:10.753508 (+ 814us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:10.753519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.755150 (+ 1631us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:10.755160 (+ 10us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:10.756647 (+ 1487us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:10.756667 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.756883 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:10.757687 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:10.757706 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:10.758342 (+ 636us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:10.762058 (+ 3716us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:10.762062 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:10.762065 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:10.762424 (+ 359us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:10.762797 (+ 373us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:10.762803 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:10.762807 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:10.762924 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:10.763480 (+ 556us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:10.763484 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:10.763630 (+ 146us) client_negotiation.cc:769] Sending connection context
1027 14:52:10.763749 (+ 119us) client_negotiation.cc:240] Negotiation successful
1027 14:52:10.764010 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":659,"thread_start_us":523,"threads_started":1}
I20251027 14:52:10.764550 8788 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:10.750196 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:46237 (local address 127.0.104.62:34959)
1027 14:52:10.750363 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:10.750369 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:10.751360 (+ 991us) server_negotiation.cc:408] Connection header received
1027 14:52:10.751617 (+ 257us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:10.751622 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:10.751702 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:10.751817 (+ 115us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:10.753713 (+ 1896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.754992 (+ 1279us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:10.756805 (+ 1813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:10.757288 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:10.758471 (+ 1183us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:10.758515 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:10.758521 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:10.758551 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:10.761811 (+ 3260us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:10.762532 (+ 721us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:10.762539 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:10.762544 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:10.762655 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:10.763027 (+ 372us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:10.763032 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:10.763034 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:10.763334 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:10.763488 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:10.763944 (+ 456us) server_negotiation.cc:300] Negotiation successful
1027 14:52:10.764201 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251027 14:52:10.772804 8764 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33363
I20251027 14:52:10.772894 8899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33363 every 8 connection(s)
I20251027 14:52:10.774077 8764 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:10.776111 8900 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34959
I20251027 14:52:10.776417 8900 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:10.777212 8900 heartbeater.cc:507] Master 127.0.104.62:34959 requested a full tablet report, sending...
I20251027 14:52:10.779306 8691 ts_manager.cc:194] Registered new tserver with Master: 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363)
I20251027 14:52:10.780975 8691 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:59119
I20251027 14:52:10.783305 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 8764
I20251027 14:52:10.783531 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:52:10.784179 416 external_mini_cluster.cc:1468] Setting key f1e0eae86ed9f37b7ec26117a4c59beb
WARNING: no policy specified for oryx/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:52:10.867339 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34959
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:11.189517 8906 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:11.189849 8906 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:11.189985 8906 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:11.198096 8906 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:11.198194 8906 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:11.198334 8906 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:52:11.198431 8906 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:11.208777 8906 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:11.210314 8906 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:11.211661 8906 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:11.220878 8912 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:11.220856 8911 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:11.221679 8914 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:11.222079 8906 server_base.cc:1047] running on GCE node
I20251027 14:52:11.222638 8906 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:11.223558 8906 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:11.224758 8906 hybrid_clock.cc:648] HybridClock initialized: now 1761576731224711 us; error 30 us; skew 500 ppm
I20251027 14:52:11.225076 8906 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:11.228340 8906 init.cc:377] Logged in from keytab as oryx/127.0.104.2@KRBTEST.COM (short username oryx)
I20251027 14:52:11.230003 8906 webserver.cc:492] Webserver started at http://127.0.104.2:45047/ using document root <none> and password file <none>
I20251027 14:52:11.230752 8906 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:11.230873 8906 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:11.231220 8906 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:11.233984 8906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "fb409e5bdffd43ffb50c754e647e7450"
format_stamp: "Formatted at 2025-10-27 14:52:11 on dist-test-slave-hxxt"
server_key: "5a86a8fabdb853e0e53a9d1eda91c107"
server_key_iv: "90d4d496162670e2cf2f2abc0982f40d"
server_key_version: "encryptionkey@0"
I20251027 14:52:11.234757 8906 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "fb409e5bdffd43ffb50c754e647e7450"
format_stamp: "Formatted at 2025-10-27 14:52:11 on dist-test-slave-hxxt"
server_key: "5a86a8fabdb853e0e53a9d1eda91c107"
server_key_iv: "90d4d496162670e2cf2f2abc0982f40d"
server_key_version: "encryptionkey@0"
I20251027 14:52:11.240273 8906 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251027 14:52:11.243925 8921 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:11.245383 8906 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20251027 14:52:11.245539 8906 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "fb409e5bdffd43ffb50c754e647e7450"
format_stamp: "Formatted at 2025-10-27 14:52:11 on dist-test-slave-hxxt"
server_key: "5a86a8fabdb853e0e53a9d1eda91c107"
server_key_iv: "90d4d496162670e2cf2f2abc0982f40d"
server_key_version: "encryptionkey@0"
I20251027 14:52:11.245738 8906 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:11.270236 8906 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:11.274525 8906 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:11.274798 8906 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:11.277805 8906 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:11.277930 8906 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:11.278045 8906 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:11.278139 8906 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:11.298000 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.278281 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:46969 (local address 127.0.104.62:34959)
1027 14:52:11.279042 (+ 761us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.279047 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.280039 (+ 992us) server_negotiation.cc:408] Connection header received
1027 14:52:11.281738 (+ 1699us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.281746 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.281842 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.281947 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:11.284287 (+ 2340us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.285727 (+ 1440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.287958 (+ 2231us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.288278 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.291326 (+ 3048us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:11.291364 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:11.291368 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:11.291397 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:11.294495 (+ 3098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.295257 (+ 762us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.295263 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.295266 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.295354 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.295804 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.295809 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.295813 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.296077 (+ 264us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:11.296215 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.297370 (+ 1155us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.297606 (+ 236us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":670,"thread_start_us":589,"threads_started":1}
I20251027 14:52:11.299360 8931 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.279094 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.104.2:46969)
1027 14:52:11.279881 (+ 787us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.279919 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.281442 (+ 1523us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.282224 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.282240 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.283012 (+ 772us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:11.284010 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.284036 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.285915 (+ 1879us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.285922 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.287218 (+ 1296us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.287237 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.287600 (+ 363us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.288586 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:11.288624 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:11.291129 (+ 2505us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:11.294623 (+ 3494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.294632 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.294651 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.295148 (+ 497us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.295455 (+ 307us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.295460 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.295464 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.295653 (+ 189us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.296291 (+ 638us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:11.296300 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:11.296860 (+ 560us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.297282 (+ 422us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.297815 (+ 533us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":537,"thread_start_us":354,"threads_started":1}
I20251027 14:52:11.317559 8906 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34545
I20251027 14:52:11.317606 9041 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34545 every 8 connection(s)
I20251027 14:52:11.319350 8906 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:11.319429 9012 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.306862 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.104.2:50035)
1027 14:52:11.307625 (+ 763us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.307648 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.307816 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.308162 (+ 346us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.308167 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.308967 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:11.309608 (+ 641us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.309620 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.310776 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.310795 (+ 19us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.312042 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.312061 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.312256 (+ 195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.313017 (+ 761us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:11.313040 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:11.314017 (+ 977us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:11.317257 (+ 3240us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.317263 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.317267 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.317659 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.317994 (+ 335us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.317997 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.317999 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.318072 (+ 73us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.318669 (+ 597us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:11.318673 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:11.318798 (+ 125us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.318907 (+ 109us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.319121 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":579,"spinlock_wait_cycles":23296,"thread_start_us":594,"threads_started":1}
I20251027 14:52:11.319602 8930 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.307396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:50035 (local address 127.0.104.62:34959)
1027 14:52:11.307555 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.307560 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.307750 (+ 190us) server_negotiation.cc:408] Connection header received
1027 14:52:11.307946 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.307950 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.308021 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.308103 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:11.309761 (+ 1658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.310599 (+ 838us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.312194 (+ 1595us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.313001 (+ 807us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.314173 (+ 1172us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:11.314215 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:11.314220 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:11.314259 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:11.317130 (+ 2871us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.317770 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.317777 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.317780 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.317879 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.318161 (+ 282us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.318166 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.318170 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.318512 (+ 342us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:11.318710 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.319032 (+ 322us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.319283 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251027 14:52:11.321700 9042 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34959
I20251027 14:52:11.322008 9042 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:11.322767 9042 heartbeater.cc:507] Master 127.0.104.62:34959 requested a full tablet report, sending...
I20251027 14:52:11.324072 8691 ts_manager.cc:194] Registered new tserver with Master: fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545)
I20251027 14:52:11.324895 8691 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:46969
I20251027 14:52:11.328936 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 8906
I20251027 14:52:11.329118 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:52:11.329458 416 external_mini_cluster.cc:1468] Setting key 70ac82d0979279cacf10b734f0bbeb2d
WARNING: no policy specified for oryx/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal oryx/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab.
I20251027 14:52:11.413043 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34959
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:34929
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--enable_txn_system_client_init=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:11.737257 9048 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:11.737576 9048 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:11.737713 9048 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:11.745860 9048 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:11.745965 9048 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:11.746105 9048 flags.cc:425] Enabled experimental flag: --enable_txn_system_client_init=true
W20251027 14:52:11.746186 9048 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:11.756661 9048 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34929
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/krb5kdc/oryx.keytab
--principal=oryx/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--enable_txn_system_client_init=true
--tserver_master_addrs=127.0.104.62:34959
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:11.758271 9048 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:11.759588 9048 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:11.768795 9053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:11.769022 9054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:11.769362 9056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:11.770112 9048 server_base.cc:1047] running on GCE node
I20251027 14:52:11.770723 9048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:11.771729 9048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:11.772950 9048 hybrid_clock.cc:648] HybridClock initialized: now 1761576731772879 us; error 61 us; skew 500 ppm
I20251027 14:52:11.773262 9048 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:11.776592 9048 init.cc:377] Logged in from keytab as oryx/127.0.104.3@KRBTEST.COM (short username oryx)
I20251027 14:52:11.778204 9048 webserver.cc:492] Webserver started at http://127.0.104.3:43089/ using document root <none> and password file <none>
I20251027 14:52:11.778903 9048 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:11.778995 9048 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:11.779294 9048 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:11.781878 9048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "384c887b5f2246c9939e7b4156550f69"
format_stamp: "Formatted at 2025-10-27 14:52:11 on dist-test-slave-hxxt"
server_key: "f27cc6bd3d340274e7a50361b7d65fdd"
server_key_iv: "86ba83b7127b3ef5848269eabf7c47a3"
server_key_version: "encryptionkey@0"
I20251027 14:52:11.782599 9048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "384c887b5f2246c9939e7b4156550f69"
format_stamp: "Formatted at 2025-10-27 14:52:11 on dist-test-slave-hxxt"
server_key: "f27cc6bd3d340274e7a50361b7d65fdd"
server_key_iv: "86ba83b7127b3ef5848269eabf7c47a3"
server_key_version: "encryptionkey@0"
I20251027 14:52:11.784202 8900 heartbeater.cc:499] Master 127.0.104.62:34959 was elected leader, sending a full tablet report...
I20251027 14:52:11.788254 9048 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251027 14:52:11.791668 9063 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:11.793120 9048 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:11.793277 9048 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "384c887b5f2246c9939e7b4156550f69"
format_stamp: "Formatted at 2025-10-27 14:52:11 on dist-test-slave-hxxt"
server_key: "f27cc6bd3d340274e7a50361b7d65fdd"
server_key_iv: "86ba83b7127b3ef5848269eabf7c47a3"
server_key_version: "encryptionkey@0"
I20251027 14:52:11.793452 9048 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:11.812628 9048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:11.816366 9048 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:11.816684 9048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:11.819516 9048 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:11.819608 9048 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:11.819787 9048 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:11.820163 9048 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576731, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:11.840978 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.820203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:49655 (local address 127.0.104.62:34959)
1027 14:52:11.820726 (+ 523us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.820733 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.821823 (+ 1090us) server_negotiation.cc:408] Connection header received
1027 14:52:11.823684 (+ 1861us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.823692 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.823794 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.823932 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:11.826146 (+ 2214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.827789 (+ 1643us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.829726 (+ 1937us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.830266 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.834008 (+ 3742us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:11.834043 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:11.834046 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:11.834089 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:11.837656 (+ 3567us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.838456 (+ 800us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.838463 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.838467 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.838575 (+ 108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.838918 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.838923 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.838927 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.839239 (+ 312us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:11.839386 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.840227 (+ 841us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.840542 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":286,"threads_started":1}
I20251027 14:52:11.841797 9073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.820786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.104.3:49655)
1027 14:52:11.821632 (+ 846us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.821678 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.823298 (+ 1620us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.824141 (+ 843us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.824150 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.824944 (+ 794us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:11.825914 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.825950 (+ 36us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.827984 (+ 2034us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.827988 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.829374 (+ 1386us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.829393 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.829788 (+ 395us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.830774 (+ 986us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:11.830809 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:11.833715 (+ 2906us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:11.837865 (+ 4150us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.837876 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.837896 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.838320 (+ 424us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.838683 (+ 363us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.838687 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.838689 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.838821 (+ 132us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.839382 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:11.839390 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:11.839788 (+ 398us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.840109 (+ 321us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.840546 (+ 437us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":545,"thread_start_us":353,"threads_started":1}
I20251027 14:52:11.861599 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.848010 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:40301 (local address 127.0.104.62:34959)
1027 14:52:11.848194 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.848201 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.848790 (+ 589us) server_negotiation.cc:408] Connection header received
1027 14:52:11.849094 (+ 304us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.849101 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.849167 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.849220 (+ 53us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:11.851211 (+ 1991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.852617 (+ 1406us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.853939 (+ 1322us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.854472 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.855895 (+ 1423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:11.855942 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:11.855947 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:11.855987 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:11.859169 (+ 3182us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.859838 (+ 669us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.859843 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.859845 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.859915 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.860211 (+ 296us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.860216 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.860220 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.860533 (+ 313us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:11.860664 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.861044 (+ 380us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.861273 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:52:11.861656 9127 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.848009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.104.3:40301)
1027 14:52:11.848673 (+ 664us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.848690 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.848916 (+ 226us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.849455 (+ 539us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.849463 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.850203 (+ 740us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:11.850968 (+ 765us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.850984 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.852809 (+ 1825us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.852814 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.853789 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.853803 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.853964 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.854713 (+ 749us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:11.854740 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:11.855686 (+ 946us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:11.859327 (+ 3641us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.859333 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.859338 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.859699 (+ 361us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.860013 (+ 314us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:11.860017 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:11.860020 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:11.860099 (+ 79us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:11.860647 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:11.860652 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:11.860807 (+ 155us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.860924 (+ 117us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.861258 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":516,"thread_start_us":356,"threads_started":1}
I20251027 14:52:11.871047 9048 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:35615
I20251027 14:52:11.871095 9183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:35615 every 8 connection(s)
I20251027 14:52:11.872488 9048 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:11.875005 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9048
I20251027 14:52:11.875111 9184 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34959
I20251027 14:52:11.875216 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:52:11.875475 9184 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:11.875710 416 external_mini_cluster.cc:1468] Setting key d856ec97171e285ecd8f294b9dfc75f7
I20251027 14:52:11.876478 9184 heartbeater.cc:507] Master 127.0.104.62:34959 requested a full tablet report, sending...
I20251027 14:52:11.878146 8691 ts_manager.cc:194] Registered new tserver with Master: 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:11.879243 8691 master_service.cc:502] Signed X509 certificate for tserver {username='oryx', principal='oryx/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:49655
I20251027 14:52:11.879429 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-user@KRBTEST.COM:
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-user@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
W20251027 14:52:11.932662 9071 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:11.919131 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39974 (local address 127.0.104.62:34959)
1027 14:52:11.919346 (+ 215us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.919352 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.919515 (+ 163us) server_negotiation.cc:408] Connection header received
1027 14:52:11.919709 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.919716 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.919797 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.919886 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:11.921389 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.922264 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.923477 (+ 1213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.924031 (+ 554us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.932332 (+ 8301us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39974: BlockingRecv error: recv got EOF from 127.0.0.1:39974 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Oct 27 14:52:11 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576731, etypes {rep=17 tkt=17 ses=17}, test-user@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:11.947576 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.935612 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39990 (local address 127.0.104.62:34959)
1027 14:52:11.935841 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.935847 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.936054 (+ 207us) server_negotiation.cc:408] Connection header received
1027 14:52:11.936179 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.936183 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.936249 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.936333 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:11.937718 (+ 1385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.938595 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.939700 (+ 1105us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.940134 (+ 434us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.943054 (+ 2920us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:11.943095 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:11.943099 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:11.943132 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:11.945598 (+ 2466us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.946186 (+ 588us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.946190 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.946193 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.946256 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:11.946469 (+ 213us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:11.946472 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:11.946474 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:11.946720 (+ 246us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:11.946889 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.947153 (+ 264us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.947320 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":99}
I20251027 14:52:11.951136 8691 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:52:11.955636 8691 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:11.990458 9204 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.967680 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33363 (local address 127.0.0.1:40188)
1027 14:52:11.980929 (+ 13249us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.980987 (+ 58us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.981258 (+ 271us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.984858 (+ 3600us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.984867 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.984905 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:11.986344 (+ 1439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.986360 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.988235 (+ 1875us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.988239 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.989447 (+ 1208us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.989459 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.989645 (+ 186us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.989718 (+ 73us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.989893 (+ 175us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.990090 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12906,"thread_start_us":16547,"threads_started":1}
I20251027 14:52:11.991752 9202 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.967980 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:34545 (local address 127.0.0.1:45340)
1027 14:52:11.984000 (+ 16020us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.984021 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.984153 (+ 132us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.986608 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.986612 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.986636 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:11.987311 (+ 675us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.987320 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.989306 (+ 1986us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.989312 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.991030 (+ 1718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.991044 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.991220 (+ 176us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.991246 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.991312 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.991431 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15901,"thread_start_us":12858,"threads_started":1}
I20251027 14:52:11.992270 9206 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.969539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:35615 (local address 127.0.0.1:42310)
1027 14:52:11.985603 (+ 16064us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:11.985620 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:11.985736 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:11.986712 (+ 976us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:11.986716 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:11.986748 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:11.987526 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.987540 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.989716 (+ 2176us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.989722 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:11.991568 (+ 1846us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:11.991582 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.991763 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.991786 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:52:11.991849 (+ 63us) client_negotiation.cc:240] Negotiation successful
1027 14:52:11.991958 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15932,"thread_start_us":15708,"threads_started":1}
I20251027 14:52:11.992471 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.967932 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40188 (local address 127.0.104.1:33363)
1027 14:52:11.983799 (+ 15867us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.983812 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.983849 (+ 37us) server_negotiation.cc:408] Connection header received
1027 14:52:11.983967 (+ 118us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.983977 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.984275 (+ 298us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.984527 (+ 252us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:11.986558 (+ 2031us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.988059 (+ 1501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.990585 (+ 2526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.991670 (+ 1085us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.991851 (+ 181us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.991976 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.992146 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15077,"thread_start_us":15507,"threads_started":1}
I20251027 14:52:11.993016 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.968333 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45340 (local address 127.0.104.2:34545)
1027 14:52:11.985779 (+ 17446us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.985788 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.985816 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:52:11.985914 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.985923 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.986179 (+ 256us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.986383 (+ 204us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:11.987486 (+ 1103us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.989105 (+ 1619us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.991212 (+ 2107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.992337 (+ 1125us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.992501 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.992589 (+ 88us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.992751 (+ 162us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":17302,"thread_start_us":17177,"threads_started":1}
I20251027 14:52:11.995003 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:11.970203 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42310 (local address 127.0.104.3:35615)
1027 14:52:11.985052 (+ 14849us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:11.985062 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:11.985810 (+ 748us) server_negotiation.cc:408] Connection header received
1027 14:52:11.985919 (+ 109us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:11.985928 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:11.986175 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:11.986383 (+ 208us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:11.987694 (+ 1311us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.989437 (+ 1743us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:11.992331 (+ 2894us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:11.993485 (+ 1154us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:11.993621 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:11.993737 (+ 116us) server_negotiation.cc:300] Negotiation successful
1027 14:52:11.993890 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14677,"thread_start_us":14511,"threads_started":1}
I20251027 14:52:11.996821 8972 tablet_service.cc:1505] Processing CreateTablet for tablet b5706641daa24262b7f268c105a23f2c (DEFAULT_TABLE table=test-table [id=2376e241b451405aaac1c78883881e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:11.996851 8830 tablet_service.cc:1505] Processing CreateTablet for tablet b5706641daa24262b7f268c105a23f2c (DEFAULT_TABLE table=test-table [id=2376e241b451405aaac1c78883881e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:11.998167 8972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5706641daa24262b7f268c105a23f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:11.998201 8830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5706641daa24262b7f268c105a23f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:11.999145 9114 tablet_service.cc:1505] Processing CreateTablet for tablet b5706641daa24262b7f268c105a23f2c (DEFAULT_TABLE table=test-table [id=2376e241b451405aaac1c78883881e5a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.000505 9114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b5706641daa24262b7f268c105a23f2c. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.008571 9208 tablet_bootstrap.cc:492] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Bootstrap starting.
I20251027 14:52:12.011729 9208 tablet_bootstrap.cc:654] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.012642 9209 tablet_bootstrap.cc:492] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Bootstrap starting.
I20251027 14:52:12.013090 9208 log.cc:826] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:12.015965 9209 tablet_bootstrap.cc:654] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.016435 9208 tablet_bootstrap.cc:492] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: No bootstrap required, opened a new log
I20251027 14:52:12.016727 9208 ts_tablet_manager.cc:1403] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Time spent bootstrapping tablet: real 0.008s user 0.008s sys 0.000s
I20251027 14:52:12.017372 9209 log.cc:826] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:12.020565 9211 tablet_bootstrap.cc:492] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Bootstrap starting.
I20251027 14:52:12.020586 9209 tablet_bootstrap.cc:492] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: No bootstrap required, opened a new log
I20251027 14:52:12.020865 9209 ts_tablet_manager.cc:1403] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251027 14:52:12.023281 9211 tablet_bootstrap.cc:654] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.023653 9208 raft_consensus.cc:359] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.024171 9208 raft_consensus.cc:385] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.024325 9208 raft_consensus.cc:740] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb409e5bdffd43ffb50c754e647e7450, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.024667 9211 log.cc:826] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:12.025154 9208 consensus_queue.cc:260] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.026486 9042 heartbeater.cc:499] Master 127.0.104.62:34959 was elected leader, sending a full tablet report...
I20251027 14:52:12.027628 9209 raft_consensus.cc:359] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.028052 9209 raft_consensus.cc:385] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.028174 9209 raft_consensus.cc:740] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.028304 9211 tablet_bootstrap.cc:492] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: No bootstrap required, opened a new log
I20251027 14:52:12.028659 9211 ts_tablet_manager.cc:1403] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20251027 14:52:12.028726 9208 ts_tablet_manager.cc:1434] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Time spent starting tablet: real 0.012s user 0.007s sys 0.003s
I20251027 14:52:12.028990 9209 consensus_queue.cc:260] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.030545 9209 ts_tablet_manager.cc:1434] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Time spent starting tablet: real 0.010s user 0.008s sys 0.000s
I20251027 14:52:12.035059 9211 raft_consensus.cc:359] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.035379 9211 raft_consensus.cc:385] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.035454 9211 raft_consensus.cc:740] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c887b5f2246c9939e7b4156550f69, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.036106 9211 consensus_queue.cc:260] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.037132 9184 heartbeater.cc:499] Master 127.0.104.62:34959 was elected leader, sending a full tablet report...
I20251027 14:52:12.037624 9211 ts_tablet_manager.cc:1434] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Time spent starting tablet: real 0.009s user 0.000s sys 0.008s
I20251027 14:52:12.037937 9215 raft_consensus.cc:493] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:12.038163 9215 raft_consensus.cc:515] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.039933 9215 leader_election.cc:290] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:12.045143 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.040151 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:35615 (local address 127.0.104.1:34751)
1027 14:52:12.040811 (+ 660us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.040829 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.040964 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.041232 (+ 268us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.041238 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.041264 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:12.041866 (+ 602us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.041875 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.043156 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.043160 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.044455 (+ 1295us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.044469 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.044636 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.044663 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.044733 (+ 70us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.044855 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"threads_started":1}
I20251027 14:52:12.045987 9207 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.040626 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:34751 (local address 127.0.104.3:35615)
1027 14:52:12.040784 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.040790 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.040898 (+ 108us) server_negotiation.cc:408] Connection header received
1027 14:52:12.041076 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.041082 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.041134 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.041203 (+ 69us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:12.042034 (+ 831us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.042955 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.044581 (+ 1626us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.045594 (+ 1013us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.045644 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.045717 (+ 73us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.045804 (+ 87us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":40}
I20251027 14:52:12.049749 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5706641daa24262b7f268c105a23f2c" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69" is_pre_election: true
I20251027 14:52:12.050064 9218 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.040178 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:34545 (local address 127.0.104.1:43591)
1027 14:52:12.045325 (+ 5147us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.045343 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.045523 (+ 180us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.045921 (+ 398us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.045926 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.045950 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:12.046330 (+ 380us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.046340 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.048015 (+ 1675us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.048019 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.049427 (+ 1408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.049439 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.049601 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.049625 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.049685 (+ 60us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.049801 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":5067,"thread_start_us":1726,"threads_started":1}
I20251027 14:52:12.050179 9139 raft_consensus.cc:2468] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 0.
I20251027 14:52:12.050879 8783 leader_election.cc:304] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, 384c887b5f2246c9939e7b4156550f69; no voters:
I20251027 14:52:12.051120 9205 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.040729 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:43591 (local address 127.0.104.2:34545)
1027 14:52:12.040893 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.040900 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.045415 (+ 4515us) server_negotiation.cc:408] Connection header received
1027 14:52:12.045705 (+ 290us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.045714 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.045799 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.045893 (+ 94us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:12.046456 (+ 563us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.047858 (+ 1402us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.049595 (+ 1737us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.050646 (+ 1051us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.050709 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.050779 (+ 70us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.050903 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:52:12.051606 9215 raft_consensus.cc:2804] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:12.051734 9215 raft_consensus.cc:493] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:12.051822 9215 raft_consensus.cc:3060] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.051942 8992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5706641daa24262b7f268c105a23f2c" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450" is_pre_election: true
I20251027 14:52:12.052356 8992 raft_consensus.cc:2468] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 0.
I20251027 14:52:12.054200 9215 raft_consensus.cc:515] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.054776 9215 leader_election.cc:290] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 election: Requested vote from peers fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:12.055140 8992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5706641daa24262b7f268c105a23f2c" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450"
I20251027 14:52:12.055234 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b5706641daa24262b7f268c105a23f2c" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69"
I20251027 14:52:12.055297 8992 raft_consensus.cc:3060] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.055414 9139 raft_consensus.cc:3060] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.057684 8992 raft_consensus.cc:2468] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 1.
I20251027 14:52:12.057680 9139 raft_consensus.cc:2468] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 1.
I20251027 14:52:12.058099 8781 leader_election.cc:304] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:12.058364 9215 raft_consensus.cc:2804] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:12.058758 9215 raft_consensus.cc:697] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Running, Role: LEADER
I20251027 14:52:12.059288 9215 consensus_queue.cc:237] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.064090 8690 catalog_manager.cc:5649] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1fa892c685384a6195fe18e0ad41f885" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } health_report { overall_health: UNKNOWN } } }
W20251027 14:52:12.071537 9043 tablet.cc:2378] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:12.098239 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.093306 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40202 (local address 127.0.104.1:33363)
1027 14:52:12.093496 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.093500 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.093519 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:52:12.093718 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.093722 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.093801 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.093925 (+ 124us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:12.094458 (+ 533us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.095354 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.096564 (+ 1210us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.096914 (+ 350us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.096997 (+ 83us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.097558 (+ 561us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.097664 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.097814 (+ 150us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.097998 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:52:12.105901 8992 raft_consensus.cc:1275] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Refusing update from remote peer 1fa892c685384a6195fe18e0ad41f885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:12.105903 9139 raft_consensus.cc:1275] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Refusing update from remote peer 1fa892c685384a6195fe18e0ad41f885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:12.106776 9219 consensus_queue.cc:1048] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:12.107009 9215 consensus_queue.cc:1048] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:12.122349 9223 mvcc.cc:204] Tried to move back new op lower bound from 7215418294696759296 to 7215418294519173120. Current Snapshot: MvccSnapshot[applied={T|T < 7215418294696759296}]
I20251027 14:52:12.130434 8690 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:12.130666 8690 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:12.134267 8690 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Sending DeleteTablet for 3 replicas of tablet b5706641daa24262b7f268c105a23f2c
I20251027 14:52:12.135437 8830 tablet_service.cc:1552] Processing DeleteTablet for tablet b5706641daa24262b7f268c105a23f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:12 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40188
I20251027 14:52:12.135437 8972 tablet_service.cc:1552] Processing DeleteTablet for tablet b5706641daa24262b7f268c105a23f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:12 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45340
I20251027 14:52:12.137018 9114 tablet_service.cc:1552] Processing DeleteTablet for tablet b5706641daa24262b7f268c105a23f2c with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:12 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:42310
I20251027 14:52:12.145506 8690 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39990:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:52:12.146379 8690 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:12.158453 8829 tablet_service.cc:1505] Processing CreateTablet for tablet 9399c5a35c1948bda59758042152aeae (DEFAULT_TABLE table=test-table [id=5b73ac2054374cc28524cb1f6ff5ab3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.158700 9113 tablet_service.cc:1505] Processing CreateTablet for tablet 9399c5a35c1948bda59758042152aeae (DEFAULT_TABLE table=test-table [id=5b73ac2054374cc28524cb1f6ff5ab3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.159004 8829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9399c5a35c1948bda59758042152aeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.159157 9113 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9399c5a35c1948bda59758042152aeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.163931 8971 tablet_service.cc:1505] Processing CreateTablet for tablet 9399c5a35c1948bda59758042152aeae (DEFAULT_TABLE table=test-table [id=5b73ac2054374cc28524cb1f6ff5ab3e]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.164574 8971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9399c5a35c1948bda59758042152aeae. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.166271 9211 tablet_bootstrap.cc:492] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Bootstrap starting.
I20251027 14:52:12.168052 9211 tablet_bootstrap.cc:654] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.177583 9208 tablet_bootstrap.cc:492] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Bootstrap starting.
I20251027 14:52:12.179351 9208 tablet_bootstrap.cc:654] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.184368 9209 tablet_bootstrap.cc:492] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Bootstrap starting.
I20251027 14:52:12.186264 9209 tablet_bootstrap.cc:654] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.191192 9234 tablet_replica.cc:333] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: stopping tablet replica
I20251027 14:52:12.193094 9234 raft_consensus.cc:2243] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:12.194703 9233 tablet_replica.cc:333] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: stopping tablet replica
I20251027 14:52:12.195968 9234 raft_consensus.cc:2272] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:12.197723 9208 tablet_bootstrap.cc:492] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: No bootstrap required, opened a new log
I20251027 14:52:12.197966 9208 ts_tablet_manager.cc:1403] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Time spent bootstrapping tablet: real 0.021s user 0.007s sys 0.001s
I20251027 14:52:12.199110 9208 raft_consensus.cc:359] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.199411 9208 raft_consensus.cc:385] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.199564 9208 raft_consensus.cc:740] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb409e5bdffd43ffb50c754e647e7450, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.199903 9208 consensus_queue.cc:260] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.200551 9208 ts_tablet_manager.cc:1434] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 14:52:12.195998 9211 tablet_bootstrap.cc:492] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: No bootstrap required, opened a new log
I20251027 14:52:12.201540 9211 ts_tablet_manager.cc:1403] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Time spent bootstrapping tablet: real 0.035s user 0.002s sys 0.016s
I20251027 14:52:12.202391 9211 raft_consensus.cc:359] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.202584 9211 raft_consensus.cc:385] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.202664 9211 raft_consensus.cc:740] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c887b5f2246c9939e7b4156550f69, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.202874 9211 consensus_queue.cc:260] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.203348 9211 ts_tablet_manager.cc:1434] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20251027 14:52:12.203871 9209 tablet_bootstrap.cc:492] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: No bootstrap required, opened a new log
I20251027 14:52:12.204016 9209 ts_tablet_manager.cc:1403] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Time spent bootstrapping tablet: real 0.020s user 0.008s sys 0.000s
I20251027 14:52:12.204847 9209 raft_consensus.cc:359] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.205574 9209 raft_consensus.cc:385] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.205683 9209 raft_consensus.cc:740] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.205897 9209 consensus_queue.cc:260] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.206377 9209 ts_tablet_manager.cc:1434] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 14:52:12.207448 9232 tablet_replica.cc:333] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: stopping tablet replica
I20251027 14:52:12.207643 9234 ts_tablet_manager.cc:1916] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:12.208681 9232 raft_consensus.cc:2243] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:12.209489 9232 raft_consensus.cc:2272] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:12.213048 9232 ts_tablet_manager.cc:1916] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:12.215132 9234 ts_tablet_manager.cc:1929] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:12.215289 9234 log.cc:1199] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/b5706641daa24262b7f268c105a23f2c
I20251027 14:52:12.215773 9234 ts_tablet_manager.cc:1950] T b5706641daa24262b7f268c105a23f2c P 384c887b5f2246c9939e7b4156550f69: Deleting consensus metadata
I20251027 14:52:12.216862 9233 raft_consensus.cc:2243] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:12.217780 9233 raft_consensus.cc:2272] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:12.219974 8678 catalog_manager.cc:4985] TS 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615): tablet b5706641daa24262b7f268c105a23f2c (table test-table [id=2376e241b451405aaac1c78883881e5a]) successfully deleted
I20251027 14:52:12.221911 9233 ts_tablet_manager.cc:1916] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:12.227475 9232 ts_tablet_manager.cc:1929] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:12.227752 9232 log.cc:1199] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/b5706641daa24262b7f268c105a23f2c
I20251027 14:52:12.228336 9232 ts_tablet_manager.cc:1950] T b5706641daa24262b7f268c105a23f2c P 1fa892c685384a6195fe18e0ad41f885: Deleting consensus metadata
I20251027 14:52:12.229172 9233 ts_tablet_manager.cc:1929] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:12.229326 9233 log.cc:1199] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/b5706641daa24262b7f268c105a23f2c
I20251027 14:52:12.229801 9233 ts_tablet_manager.cc:1950] T b5706641daa24262b7f268c105a23f2c P fb409e5bdffd43ffb50c754e647e7450: Deleting consensus metadata
I20251027 14:52:12.230232 8677 catalog_manager.cc:4985] TS 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363): tablet b5706641daa24262b7f268c105a23f2c (table test-table [id=2376e241b451405aaac1c78883881e5a]) successfully deleted
I20251027 14:52:12.231295 8676 catalog_manager.cc:4985] TS fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545): tablet b5706641daa24262b7f268c105a23f2c (table test-table [id=2376e241b451405aaac1c78883881e5a]) successfully deleted
W20251027 14:52:12.276451 8901 tablet.cc:2378] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:12.288416 9219 raft_consensus.cc:493] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:12.288657 9219 raft_consensus.cc:515] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.289319 9219 leader_election.cc:290] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:12.289844 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9399c5a35c1948bda59758042152aeae" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69" is_pre_election: true
I20251027 14:52:12.289860 8992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9399c5a35c1948bda59758042152aeae" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450" is_pre_election: true
I20251027 14:52:12.290048 9139 raft_consensus.cc:2468] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 0.
I20251027 14:52:12.290083 8992 raft_consensus.cc:2468] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 0.
I20251027 14:52:12.290444 8783 leader_election.cc:304] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, 384c887b5f2246c9939e7b4156550f69; no voters:
I20251027 14:52:12.290735 9219 raft_consensus.cc:2804] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:12.290831 9219 raft_consensus.cc:493] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:12.290940 9219 raft_consensus.cc:3060] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.293381 9219 raft_consensus.cc:515] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.293964 9219 leader_election.cc:290] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 election: Requested vote from peers fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:12.294369 8992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9399c5a35c1948bda59758042152aeae" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450"
I20251027 14:52:12.294471 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "9399c5a35c1948bda59758042152aeae" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69"
I20251027 14:52:12.294535 8992 raft_consensus.cc:3060] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.294664 9139 raft_consensus.cc:3060] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.296602 8992 raft_consensus.cc:2468] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 1.
I20251027 14:52:12.296969 8781 leader_election.cc:304] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:12.297210 9219 raft_consensus.cc:2804] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:12.297380 9219 raft_consensus.cc:697] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Running, Role: LEADER
I20251027 14:52:12.297711 9139 raft_consensus.cc:2468] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 1.
I20251027 14:52:12.297711 9219 consensus_queue.cc:237] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.300523 8690 catalog_manager.cc:5649] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1fa892c685384a6195fe18e0ad41f885" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } health_report { overall_health: UNKNOWN } } }
Oct 27 14:52:12 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576730, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.62@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:12.349093 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.335609 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34959 (local address 127.0.0.1:40004)
1027 14:52:12.336184 (+ 575us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.336210 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.336359 (+ 149us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.336811 (+ 452us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.336817 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.337544 (+ 727us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:12.337980 (+ 436us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.337990 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.339624 (+ 1634us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.339628 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.340428 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.340441 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.340580 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.341240 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:12.341262 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:12.343323 (+ 2061us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:12.346934 (+ 3611us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:12.346945 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:12.346950 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:12.347317 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:12.347668 (+ 351us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:12.347672 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:12.347674 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:12.347762 (+ 88us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:12.348263 (+ 501us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:12.348270 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:12.348466 (+ 196us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.348604 (+ 138us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.348832 (+ 228us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":448,"thread_start_us":335,"threads_started":1}
I20251027 14:52:12.349390 9071 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.335559 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40004 (local address 127.0.104.62:34959)
1027 14:52:12.335738 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.335744 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.336306 (+ 562us) server_negotiation.cc:408] Connection header received
1027 14:52:12.336541 (+ 235us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.336548 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.336630 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.336765 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:12.338124 (+ 1359us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.339476 (+ 1352us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.340551 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.341013 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.343456 (+ 2443us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:12.343496 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:12.343502 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:12.343535 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:12.346791 (+ 3256us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:12.347458 (+ 667us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:12.347465 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:12.347468 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:12.347550 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:12.347860 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:12.347865 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:12.347868 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:12.348138 (+ 270us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:12.348275 (+ 137us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.348710 (+ 435us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.348981 (+ 271us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":28,"server-negotiator.queue_time_us":66}
I20251027 14:52:12.354182 8690 catalog_manager.cc:2257] Servicing CreateTable request from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40004:
name: "kudu_system.kudu_transactions"
schema {
columns {
name: "txn_id"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "entry_type"
type: INT8
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "identifier"
type: STRING
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "metadata"
type: STRING
is_key: false
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
rows: "<redacted>""\006\001\000\000\000\000\000\000\000\000\007\001@B\017\000\000\000\000\000"
indirect_data: "<redacted>"""
}
partition_schema {
range_schema {
columns {
name: "txn_id"
}
}
}
table_type: TXN_STATUS_TABLE
W20251027 14:52:12.355209 8690 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table kudu_system.kudu_transactions in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:12.362716 8972 tablet_service.cc:1505] Processing CreateTablet for tablet 99f6ec335fb044c58dae9febd47f60a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06204af7d4f44390836e0b67cf538149]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:52:12.363157 8830 tablet_service.cc:1505] Processing CreateTablet for tablet 99f6ec335fb044c58dae9febd47f60a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06204af7d4f44390836e0b67cf538149]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:52:12.363340 8972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f6ec335fb044c58dae9febd47f60a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.363727 8830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f6ec335fb044c58dae9febd47f60a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.364902 9114 tablet_service.cc:1505] Processing CreateTablet for tablet 99f6ec335fb044c58dae9febd47f60a4 (TXN_STATUS_TABLE table=kudu_system.kudu_transactions [id=06204af7d4f44390836e0b67cf538149]), partition=RANGE (txn_id) PARTITION 0 <= VALUES < 1000000
I20251027 14:52:12.365371 9114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 99f6ec335fb044c58dae9febd47f60a4. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.371045 9208 tablet_bootstrap.cc:492] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: Bootstrap starting.
I20251027 14:52:12.371766 9209 tablet_bootstrap.cc:492] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: Bootstrap starting.
I20251027 14:52:12.372774 9208 tablet_bootstrap.cc:654] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.373127 9211 tablet_bootstrap.cc:492] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: Bootstrap starting.
I20251027 14:52:12.373478 9209 tablet_bootstrap.cc:654] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.374707 9211 tablet_bootstrap.cc:654] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: Neither blocks nor log segments found. Creating new log.
W20251027 14:52:12.375299 9185 tablet.cc:2378] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:12.381702 9211 tablet_bootstrap.cc:492] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: No bootstrap required, opened a new log
I20251027 14:52:12.381970 9211 ts_tablet_manager.cc:1403] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251027 14:52:12.382766 9211 raft_consensus.cc:359] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.383038 9211 raft_consensus.cc:385] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.383140 9211 raft_consensus.cc:740] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c887b5f2246c9939e7b4156550f69, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.383473 9208 tablet_bootstrap.cc:492] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: No bootstrap required, opened a new log
I20251027 14:52:12.383432 9211 consensus_queue.cc:260] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.383770 9208 ts_tablet_manager.cc:1403] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.008s
I20251027 14:52:12.383841 9216 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.384040 9216 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.384543 9208 raft_consensus.cc:359] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.384752 9208 raft_consensus.cc:385] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.384819 9208 raft_consensus.cc:740] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb409e5bdffd43ffb50c754e647e7450, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.385150 9208 consensus_queue.cc:260] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.385571 9214 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.385864 9214 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.386286 9208 ts_tablet_manager.cc:1434] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20251027 14:52:12.387147 9209 tablet_bootstrap.cc:492] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: No bootstrap required, opened a new log
I20251027 14:52:12.387393 9209 ts_tablet_manager.cc:1403] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.009s
I20251027 14:52:12.388157 9209 raft_consensus.cc:359] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.388331 9209 raft_consensus.cc:385] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.388434 9209 raft_consensus.cc:740] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.388674 9209 consensus_queue.cc:260] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.389127 9209 ts_tablet_manager.cc:1434] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 14:52:12.389151 9219 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: TxnStatusTablet state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.389783 9219 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.391279 9211 ts_tablet_manager.cc:1434] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: Time spent starting tablet: real 0.009s user 0.000s sys 0.001s
I20251027 14:52:12.585114 9216 raft_consensus.cc:493] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:12.585328 9216 raft_consensus.cc:515] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.586799 9216 leader_election.cc:290] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363), fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545)
I20251027 14:52:12.603355 9248 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.587063 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33363 (local address 127.0.104.3:45375)
1027 14:52:12.597438 (+ 10375us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.597470 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.597706 (+ 236us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.598499 (+ 793us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.598504 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.598530 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:12.599158 (+ 628us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.599174 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.600913 (+ 1739us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.600918 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.602562 (+ 1644us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.602571 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.602725 (+ 154us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.602798 (+ 73us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.602870 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.603013 (+ 143us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10220,"thread_start_us":10093,"threads_started":1}
I20251027 14:52:12.603399 9249 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.587108 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:34545 (local address 127.0.104.3:51529)
1027 14:52:12.597784 (+ 10676us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.597804 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.597959 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.598866 (+ 907us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.598871 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.598894 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:12.599466 (+ 572us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.599475 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.600774 (+ 1299us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.600778 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.602532 (+ 1754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.602550 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.602770 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.602797 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.602871 (+ 74us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.603013 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10547,"thread_start_us":10677,"threads_started":1}
I20251027 14:52:12.604328 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.587422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:45375 (local address 127.0.104.1:33363)
1027 14:52:12.587622 (+ 200us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.587629 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.597598 (+ 9969us) server_negotiation.cc:408] Connection header received
1027 14:52:12.598162 (+ 564us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.598170 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.598252 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.598311 (+ 59us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:12.599339 (+ 1028us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.600727 (+ 1388us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.602729 (+ 2002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.603819 (+ 1090us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.603867 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.603953 (+ 86us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.604079 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251027 14:52:12.604347 9250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.587789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:51529 (local address 127.0.104.2:34545)
1027 14:52:12.598353 (+ 10564us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.598360 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.598379 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:52:12.598468 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.598474 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.598543 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.598743 (+ 200us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:12.599641 (+ 898us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.600607 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.602736 (+ 2129us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.603813 (+ 1077us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.603868 (+ 55us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.603953 (+ 85us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.604080 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10447,"thread_start_us":10359,"threads_started":1}
I20251027 14:52:12.605075 8992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99f6ec335fb044c58dae9febd47f60a4" candidate_uuid: "384c887b5f2246c9939e7b4156550f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450" is_pre_election: true
I20251027 14:52:12.605088 8855 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99f6ec335fb044c58dae9febd47f60a4" candidate_uuid: "384c887b5f2246c9939e7b4156550f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa892c685384a6195fe18e0ad41f885" is_pre_election: true
I20251027 14:52:12.605285 8992 raft_consensus.cc:2468] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384c887b5f2246c9939e7b4156550f69 in term 0.
I20251027 14:52:12.605434 8855 raft_consensus.cc:2468] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 384c887b5f2246c9939e7b4156550f69 in term 0.
I20251027 14:52:12.605883 9065 leader_election.cc:304] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384c887b5f2246c9939e7b4156550f69, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:12.606258 9216 raft_consensus.cc:2804] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:12.606389 9216 raft_consensus.cc:493] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:12.606484 9216 raft_consensus.cc:3060] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.608749 9216 raft_consensus.cc:515] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.609345 9216 leader_election.cc:290] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [CANDIDATE]: Term 1 election: Requested vote from peers 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363), fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545)
I20251027 14:52:12.609820 8992 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99f6ec335fb044c58dae9febd47f60a4" candidate_uuid: "384c887b5f2246c9939e7b4156550f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450"
I20251027 14:52:12.609838 8855 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "99f6ec335fb044c58dae9febd47f60a4" candidate_uuid: "384c887b5f2246c9939e7b4156550f69" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa892c685384a6195fe18e0ad41f885"
I20251027 14:52:12.609992 8855 raft_consensus.cc:3060] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.610001 8992 raft_consensus.cc:3060] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:12.612046 8992 raft_consensus.cc:2468] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384c887b5f2246c9939e7b4156550f69 in term 1.
I20251027 14:52:12.612198 8855 raft_consensus.cc:2468] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 384c887b5f2246c9939e7b4156550f69 in term 1.
I20251027 14:52:12.612542 9065 leader_election.cc:304] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384c887b5f2246c9939e7b4156550f69, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:12.612849 9216 raft_consensus.cc:2804] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:12.613194 9216 raft_consensus.cc:697] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [term 1 LEADER]: Becoming Leader. State: Replica: 384c887b5f2246c9939e7b4156550f69, State: Running, Role: LEADER
I20251027 14:52:12.613626 9216 consensus_queue.cc:237] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.615468 9251 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: TxnStatusTablet state changed. Reason: New leader 384c887b5f2246c9939e7b4156550f69. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.615813 9251 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:52:12.616544 9253 txn_status_manager.cc:874] Waiting until node catch up with all replicated operations in previous term...
I20251027 14:52:12.616775 9253 txn_status_manager.cc:930] Loading transaction status metadata into memory...
I20251027 14:52:12.617229 8690 catalog_manager.cc:5649] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 reported cstate change: term changed from 0 to 1, leader changed from <none> to 384c887b5f2246c9939e7b4156550f69 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } health_report { overall_health: HEALTHY } } }
I20251027 14:52:12.687462 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.681585 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.0.104.3:35615)
1027 14:52:12.682023 (+ 438us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.682027 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.682042 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:52:12.682124 (+ 82us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.682128 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.682210 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.682334 (+ 124us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:12.683025 (+ 691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.684209 (+ 1184us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.685798 (+ 1589us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.686143 (+ 345us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.686215 (+ 72us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.686845 (+ 630us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.686966 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.687122 (+ 156us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.687203 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":346,"thread_start_us":233,"threads_started":1}
I20251027 14:52:12.687511 9244 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.681401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:35615 (local address 127.0.0.1:42314)
1027 14:52:12.681611 (+ 210us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.681632 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.681820 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.682365 (+ 545us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.682371 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.682384 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:52:12.682841 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.682853 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.684349 (+ 1496us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.684355 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.685649 (+ 1294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.685665 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.685814 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.685878 (+ 64us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.686963 (+ 1085us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.686983 (+ 20us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.687071 (+ 88us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.687211 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":51}
I20251027 14:52:12.692028 8992 raft_consensus.cc:1275] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Refusing update from remote peer 384c887b5f2246c9939e7b4156550f69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:12.692224 8855 raft_consensus.cc:1275] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Refusing update from remote peer 384c887b5f2246c9939e7b4156550f69: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:12.692824 9251 consensus_queue.cc:1048] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:12.693078 9216 consensus_queue.cc:1048] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:12.705793 9219 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: TxnStatusTablet state changed. Reason: New leader 384c887b5f2246c9939e7b4156550f69. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.706040 9219 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.708716 9216 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.708956 9216 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:52:12.711453 9219 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.711658 9219 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 1fa892c685384a6195fe18e0ad41f885: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.712293 9259 mvcc.cc:204] Tried to move back new op lower bound from 7215418297099714560 to 7215418296789188608. Current Snapshot: MvccSnapshot[applied={T|T < 7215418297099714560 or (T in {7215418297099714560})}]
I20251027 14:52:12.714058 9214 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: TxnStatusTablet state changed. Reason: New leader 384c887b5f2246c9939e7b4156550f69. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.714277 9214 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.714715 9251 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: TxnStatusTablet state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.714938 9251 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P 384c887b5f2246c9939e7b4156550f69: This TxnStatusTablet replica's current role is: LEADER
I20251027 14:52:12.718281 9214 tablet_replica.cc:442] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: TxnStatusTablet state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "384c887b5f2246c9939e7b4156550f69" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } } }
I20251027 14:52:12.718463 9214 tablet_replica.cc:445] T 99f6ec335fb044c58dae9febd47f60a4 P fb409e5bdffd43ffb50c754e647e7450: This TxnStatusTablet replica's current role is: FOLLOWER
I20251027 14:52:12.720335 9219 consensus_queue.cc:1048] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:12.728744 9219 consensus_queue.cc:1048] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:12.736460 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.729524 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:45215 (local address 127.0.104.3:35615)
1027 14:52:12.729695 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.729702 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.730344 (+ 642us) server_negotiation.cc:408] Connection header received
1027 14:52:12.730614 (+ 270us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.730620 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.730707 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.730805 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:12.731802 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.733126 (+ 1324us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.734838 (+ 1712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.735274 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.735654 (+ 380us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.735822 (+ 168us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.735885 (+ 63us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.736089 (+ 204us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.736172 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:52:12.736961 9270 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.729547 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:35615 (local address 127.0.104.1:45215)
1027 14:52:12.730230 (+ 683us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.730256 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.730465 (+ 209us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.731001 (+ 536us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.731007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.731020 (+ 13us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:52:12.731618 (+ 598us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.731634 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.733321 (+ 1687us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.733327 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.734695 (+ 1368us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.734709 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.735425 (+ 716us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.735488 (+ 63us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.735964 (+ 476us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.735973 (+ 9us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.736620 (+ 647us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.736745 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":1437,"threads_started":1}
I20251027 14:52:12.781006 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.770747 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:48643 (local address 127.0.104.1:33363)
1027 14:52:12.770900 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.770907 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.775251 (+ 4344us) server_negotiation.cc:408] Connection header received
1027 14:52:12.775434 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.775440 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.775541 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.775626 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:12.776688 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.777988 (+ 1300us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.779645 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.780155 (+ 510us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.780237 (+ 82us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.780401 (+ 164us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.780492 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.780644 (+ 152us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.780744 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251027 14:52:12.781116 9274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.770648 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33363 (local address 127.0.104.3:48643)
1027 14:52:12.775122 (+ 4474us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:12.775153 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:12.775314 (+ 161us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:12.775738 (+ 424us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:12.775744 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:12.775758 (+ 14us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:52:12.776522 (+ 764us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.776536 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.778151 (+ 1615us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.778158 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:12.779510 (+ 1352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:12.779525 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.779703 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.779773 (+ 70us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:52:12.780517 (+ 744us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:52:12.780530 (+ 13us) client_negotiation.cc:769] Sending connection context
1027 14:52:12.780625 (+ 95us) client_negotiation.cc:240] Negotiation successful
1027 14:52:12.780769 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4262,"thread_start_us":4098,"threads_started":1}
I20251027 14:52:12.820567 8691 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:12.820776 8691 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-user', principal='test-user@KRBTEST.COM'} at 127.0.0.1:39990:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:12.824137 8691 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Sending DeleteTablet for 3 replicas of tablet 9399c5a35c1948bda59758042152aeae
I20251027 14:52:12.824949 8830 tablet_service.cc:1552] Processing DeleteTablet for tablet 9399c5a35c1948bda59758042152aeae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:12 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40188
I20251027 14:52:12.825105 8972 tablet_service.cc:1552] Processing DeleteTablet for tablet 9399c5a35c1948bda59758042152aeae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:12 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45340
I20251027 14:52:12.825625 9276 tablet_replica.cc:333] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: stopping tablet replica
I20251027 14:52:12.825649 9114 tablet_service.cc:1552] Processing DeleteTablet for tablet 9399c5a35c1948bda59758042152aeae with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:12 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:42310
I20251027 14:52:12.825806 9276 raft_consensus.cc:2243] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:12.826212 9276 raft_consensus.cc:2272] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:12.826380 9278 tablet_replica.cc:333] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: stopping tablet replica
I20251027 14:52:12.826932 9278 raft_consensus.cc:2243] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:12.827193 9276 ts_tablet_manager.cc:1916] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:12.827248 9278 raft_consensus.cc:2272] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:12.832644 9276 ts_tablet_manager.cc:1929] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:52:12.832859 9276 log.cc:1199] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/9399c5a35c1948bda59758042152aeae
I20251027 14:52:12.833261 9276 ts_tablet_manager.cc:1950] T 9399c5a35c1948bda59758042152aeae P 1fa892c685384a6195fe18e0ad41f885: Deleting consensus metadata
I20251027 14:52:12.834348 8677 catalog_manager.cc:4985] TS 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363): tablet 9399c5a35c1948bda59758042152aeae (table test-table [id=5b73ac2054374cc28524cb1f6ff5ab3e]) successfully deleted
I20251027 14:52:12.840914 9278 ts_tablet_manager.cc:1916] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:12.848618 9277 tablet_replica.cc:333] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: stopping tablet replica
I20251027 14:52:12.848762 9278 ts_tablet_manager.cc:1929] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:52:12.848961 9278 log.cc:1199] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/9399c5a35c1948bda59758042152aeae
I20251027 14:52:12.848995 9277 raft_consensus.cc:2243] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:12.849278 9277 raft_consensus.cc:2272] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:12.849534 9278 ts_tablet_manager.cc:1950] T 9399c5a35c1948bda59758042152aeae P 384c887b5f2246c9939e7b4156550f69: Deleting consensus metadata
I20251027 14:52:12.850689 9277 ts_tablet_manager.cc:1916] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:12.851117 8678 catalog_manager.cc:4985] TS 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615): tablet 9399c5a35c1948bda59758042152aeae (table test-table [id=5b73ac2054374cc28524cb1f6ff5ab3e]) successfully deleted
I20251027 14:52:12.856061 9277 ts_tablet_manager.cc:1929] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:52:12.856184 9277 log.cc:1199] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/9399c5a35c1948bda59758042152aeae
I20251027 14:52:12.856642 9277 ts_tablet_manager.cc:1950] T 9399c5a35c1948bda59758042152aeae P fb409e5bdffd43ffb50c754e647e7450: Deleting consensus metadata
I20251027 14:52:12.857724 8676 catalog_manager.cc:4985] TS fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545): tablet 9399c5a35c1948bda59758042152aeae (table test-table [id=5b73ac2054374cc28524cb1f6ff5ab3e]) successfully deleted
Oct 27 14:52:12 dist-test-slave-hxxt krb5kdc[8643](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Oct 27 14:52:12 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:12 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
W20251027 14:52:12.902454 9286 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:12.883081 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40008 (local address 127.0.104.62:34959)
1027 14:52:12.893489 (+ 10408us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.893495 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.893511 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:52:12.893561 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.893567 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.893653 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.893771 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:12.895331 (+ 1560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.896267 (+ 936us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.897462 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.897922 (+ 460us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.902199 (+ 4277us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40008: BlockingRecv error: recv got EOF from 127.0.0.1:40008 (error 108)
Metrics: {"server-negotiator.queue_time_us":10296,"thread_start_us":10173,"threads_started":1}
Oct 27 14:52:12 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM
I20251027 14:52:12.922288 9286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:12.909792 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40024 (local address 127.0.104.62:34959)
1027 14:52:12.909962 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:12.909969 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:12.910409 (+ 440us) server_negotiation.cc:408] Connection header received
1027 14:52:12.910563 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:12.910568 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:12.910636 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:12.910716 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:12.912057 (+ 1341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.912971 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:12.914089 (+ 1118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:12.914519 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:12.917537 (+ 3018us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:12.917564 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:12.917568 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:12.917598 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:12.920103 (+ 2505us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:12.920725 (+ 622us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:12.920730 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:12.920732 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:12.920807 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:12.921072 (+ 265us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:12.921077 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:12.921080 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:12.921301 (+ 221us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:12.921523 (+ 222us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:12.921763 (+ 240us) server_negotiation.cc:300] Negotiation successful
1027 14:52:12.921980 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:52:12.925518 8691 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:52:12.926084 8691 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:12.932621 8830 tablet_service.cc:1505] Processing CreateTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 (DEFAULT_TABLE table=test-table [id=04c61663d82745f6b883e41a61ba4e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.933063 8830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28462051a1d747ad8e62ed9b61985ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.933095 8972 tablet_service.cc:1505] Processing CreateTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 (DEFAULT_TABLE table=test-table [id=04c61663d82745f6b883e41a61ba4e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.933401 8972 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28462051a1d747ad8e62ed9b61985ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.936813 9114 tablet_service.cc:1505] Processing CreateTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 (DEFAULT_TABLE table=test-table [id=04c61663d82745f6b883e41a61ba4e8b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:12.937196 9114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 28462051a1d747ad8e62ed9b61985ad0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:12.947728 9293 tablet_bootstrap.cc:492] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Bootstrap starting.
I20251027 14:52:12.948344 9292 tablet_bootstrap.cc:492] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Bootstrap starting.
I20251027 14:52:12.950125 9293 tablet_bootstrap.cc:654] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.950829 9292 tablet_bootstrap.cc:654] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.958516 9293 tablet_bootstrap.cc:492] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: No bootstrap required, opened a new log
I20251027 14:52:12.958786 9293 ts_tablet_manager.cc:1403] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Time spent bootstrapping tablet: real 0.011s user 0.010s sys 0.000s
I20251027 14:52:12.959651 9293 raft_consensus.cc:359] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.959968 9293 raft_consensus.cc:385] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.960098 9293 raft_consensus.cc:740] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.960467 9293 consensus_queue.cc:260] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.960878 9294 tablet_bootstrap.cc:492] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Bootstrap starting.
I20251027 14:52:12.962963 9294 tablet_bootstrap.cc:654] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:12.963528 9293 ts_tablet_manager.cc:1434] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20251027 14:52:12.967242 9292 tablet_bootstrap.cc:492] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: No bootstrap required, opened a new log
I20251027 14:52:12.967419 9292 ts_tablet_manager.cc:1403] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Time spent bootstrapping tablet: real 0.019s user 0.014s sys 0.003s
I20251027 14:52:12.968220 9292 raft_consensus.cc:359] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.968487 9292 raft_consensus.cc:385] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.968601 9292 raft_consensus.cc:740] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb409e5bdffd43ffb50c754e647e7450, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.968845 9292 consensus_queue.cc:260] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.969956 9292 ts_tablet_manager.cc:1434] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20251027 14:52:12.971546 9294 tablet_bootstrap.cc:492] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: No bootstrap required, opened a new log
I20251027 14:52:12.971706 9294 ts_tablet_manager.cc:1403] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20251027 14:52:12.972554 9294 raft_consensus.cc:359] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.972851 9294 raft_consensus.cc:385] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:12.972999 9294 raft_consensus.cc:740] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c887b5f2246c9939e7b4156550f69, State: Initialized, Role: FOLLOWER
I20251027 14:52:12.973304 9294 consensus_queue.cc:260] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:12.973978 9294 ts_tablet_manager.cc:1434] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20251027 14:52:13.019740 9272 raft_consensus.cc:493] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:13.019963 9272 raft_consensus.cc:515] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.020829 9272 leader_election.cc:290] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:13.021378 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28462051a1d747ad8e62ed9b61985ad0" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69" is_pre_election: true
I20251027 14:52:13.021397 8991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28462051a1d747ad8e62ed9b61985ad0" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450" is_pre_election: true
I20251027 14:52:13.021636 8991 raft_consensus.cc:2468] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 0.
I20251027 14:52:13.021659 9139 raft_consensus.cc:2468] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 0.
I20251027 14:52:13.022127 8781 leader_election.cc:304] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:13.022500 9272 raft_consensus.cc:2804] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:13.022625 9272 raft_consensus.cc:493] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:13.022732 9272 raft_consensus.cc:3060] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:13.024894 9272 raft_consensus.cc:515] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.025434 9272 leader_election.cc:290] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 election: Requested vote from peers fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:13.025878 8991 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28462051a1d747ad8e62ed9b61985ad0" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fb409e5bdffd43ffb50c754e647e7450"
I20251027 14:52:13.025965 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "28462051a1d747ad8e62ed9b61985ad0" candidate_uuid: "1fa892c685384a6195fe18e0ad41f885" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69"
I20251027 14:52:13.026067 8991 raft_consensus.cc:3060] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:13.026168 9139 raft_consensus.cc:3060] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:13.029247 8991 raft_consensus.cc:2468] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 1.
I20251027 14:52:13.029381 9139 raft_consensus.cc:2468] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 1fa892c685384a6195fe18e0ad41f885 in term 1.
I20251027 14:52:13.029731 8781 leader_election.cc:304] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:13.030037 9272 raft_consensus.cc:2804] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:13.030180 9272 raft_consensus.cc:697] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 1 LEADER]: Becoming Leader. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Running, Role: LEADER
I20251027 14:52:13.030469 9272 consensus_queue.cc:237] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.033310 8691 catalog_manager.cc:5649] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 reported cstate change: term changed from 0 to 1, leader changed from <none> to 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "1fa892c685384a6195fe18e0ad41f885" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20251027 14:52:13.051411 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.047024 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40208 (local address 127.0.104.1:33363)
1027 14:52:13.047175 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.047181 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.047204 (+ 23us) server_negotiation.cc:408] Connection header received
1027 14:52:13.047471 (+ 267us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.047476 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.047544 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.047622 (+ 78us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:13.048170 (+ 548us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.049055 (+ 885us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.050251 (+ 1196us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.050648 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.050716 (+ 68us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:13.050853 (+ 137us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:13.050932 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.051067 (+ 135us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.051159 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:52:13.054076 8991 raft_consensus.cc:1275] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Refusing update from remote peer 1fa892c685384a6195fe18e0ad41f885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:13.054111 9139 raft_consensus.cc:1275] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Refusing update from remote peer 1fa892c685384a6195fe18e0ad41f885: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:13.054687 9219 consensus_queue.cc:1048] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:13.054909 9272 consensus_queue.cc:1048] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:13.068782 8688 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40024:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:13.069015 8688 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40024:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:13.071000 9258 mvcc.cc:204] Tried to move back new op lower bound from 7215418298584006656 to 7215418298495004672. Current Snapshot: MvccSnapshot[applied={T|T < 7215418298584006656}]
I20251027 14:52:13.073241 8688 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Sending DeleteTablet for 3 replicas of tablet 28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.074235 9114 tablet_service.cc:1552] Processing DeleteTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:42310
I20251027 14:52:13.074297 8972 tablet_service.cc:1552] Processing DeleteTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45340
I20251027 14:52:13.074515 9278 tablet_replica.cc:333] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: stopping tablet replica
I20251027 14:52:13.074550 8830 tablet_service.cc:1552] Processing DeleteTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40188
I20251027 14:52:13.074718 9278 raft_consensus.cc:2243] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:13.074811 8971 tablet_service.cc:1552] Processing DeleteTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45340
I20251027 14:52:13.074929 9278 raft_consensus.cc:2272] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:13.075035 8829 tablet_service.cc:1552] Processing DeleteTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40188
I20251027 14:52:13.075181 9277 tablet_replica.cc:333] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: stopping tablet replica
I20251027 14:52:13.075260 9276 tablet_replica.cc:333] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: stopping tablet replica
I20251027 14:52:13.075359 9277 raft_consensus.cc:2243] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:13.075443 9276 raft_consensus.cc:2243] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:13.075544 9277 raft_consensus.cc:2272] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:13.075733 9276 raft_consensus.cc:2272] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:13.076303 9278 ts_tablet_manager.cc:1916] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:13.076579 9277 ts_tablet_manager.cc:1916] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:13.078182 9276 ts_tablet_manager.cc:1916] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:13.083149 9277 ts_tablet_manager.cc:1929] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:13.083272 9277 log.cc:1199] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.083287 9278 ts_tablet_manager.cc:1929] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:13.083420 9278 log.cc:1199] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.083639 9277 ts_tablet_manager.cc:1950] T 28462051a1d747ad8e62ed9b61985ad0 P fb409e5bdffd43ffb50c754e647e7450: Deleting consensus metadata
I20251027 14:52:13.083813 9278 ts_tablet_manager.cc:1950] T 28462051a1d747ad8e62ed9b61985ad0 P 384c887b5f2246c9939e7b4156550f69: Deleting consensus metadata
I20251027 14:52:13.084661 8688 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40024:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
I20251027 14:52:13.084821 9276 ts_tablet_manager.cc:1929] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:13.084944 9276 log.cc:1199] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.084894 8676 catalog_manager.cc:4985] TS fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545): tablet 28462051a1d747ad8e62ed9b61985ad0 (table test-table [id=04c61663d82745f6b883e41a61ba4e8b]) successfully deleted
I20251027 14:52:13.085289 9276 ts_tablet_manager.cc:1950] T 28462051a1d747ad8e62ed9b61985ad0 P 1fa892c685384a6195fe18e0ad41f885: Deleting consensus metadata
W20251027 14:52:13.085503 8688 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
W20251027 14:52:13.086032 8676 catalog_manager.cc:4952] TS fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545): delete failed for tablet 28462051a1d747ad8e62ed9b61985ad0 because the tablet was not found. No further retry: Not found: Tablet not found: 28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.086287 8677 catalog_manager.cc:4985] TS 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363): tablet 28462051a1d747ad8e62ed9b61985ad0 (table test-table [id=04c61663d82745f6b883e41a61ba4e8b]) successfully deleted
W20251027 14:52:13.087034 8677 catalog_manager.cc:4952] TS 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363): delete failed for tablet 28462051a1d747ad8e62ed9b61985ad0 because the tablet was not found. No further retry: Not found: Tablet not found: 28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.087333 8678 catalog_manager.cc:4985] TS 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615): tablet 28462051a1d747ad8e62ed9b61985ad0 (table test-table [id=04c61663d82745f6b883e41a61ba4e8b]) successfully deleted
I20251027 14:52:13.089458 8971 tablet_service.cc:1552] Processing DeleteTablet for tablet 28462051a1d747ad8e62ed9b61985ad0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45340
W20251027 14:52:13.090039 8676 catalog_manager.cc:4952] TS fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545): delete failed for tablet 28462051a1d747ad8e62ed9b61985ad0 because the tablet was not found. No further retry: Not found: Tablet not found: 28462051a1d747ad8e62ed9b61985ad0
I20251027 14:52:13.092519 8971 tablet_service.cc:1505] Processing CreateTablet for tablet 899c85be64064ef1b9aa16765148b9e3 (DEFAULT_TABLE table=test-table [id=a9c386befd614377ba42b28258073e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:13.092947 8971 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899c85be64064ef1b9aa16765148b9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:13.093044 8829 tablet_service.cc:1505] Processing CreateTablet for tablet 899c85be64064ef1b9aa16765148b9e3 (DEFAULT_TABLE table=test-table [id=a9c386befd614377ba42b28258073e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:13.093416 8829 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899c85be64064ef1b9aa16765148b9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:13.098433 9114 tablet_service.cc:1505] Processing CreateTablet for tablet 899c85be64064ef1b9aa16765148b9e3 (DEFAULT_TABLE table=test-table [id=a9c386befd614377ba42b28258073e87]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:13.098843 9114 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 899c85be64064ef1b9aa16765148b9e3. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:13.101502 9293 tablet_bootstrap.cc:492] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Bootstrap starting.
I20251027 14:52:13.102880 9293 tablet_bootstrap.cc:654] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:13.104005 9294 tablet_bootstrap.cc:492] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Bootstrap starting.
I20251027 14:52:13.104683 9293 tablet_bootstrap.cc:492] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: No bootstrap required, opened a new log
I20251027 14:52:13.104840 9293 ts_tablet_manager.cc:1403] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Time spent bootstrapping tablet: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:13.105262 9292 tablet_bootstrap.cc:492] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Bootstrap starting.
I20251027 14:52:13.105572 9294 tablet_bootstrap.cc:654] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:13.105948 9293 raft_consensus.cc:359] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.106132 9293 raft_consensus.cc:385] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:13.106196 9293 raft_consensus.cc:740] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1fa892c685384a6195fe18e0ad41f885, State: Initialized, Role: FOLLOWER
I20251027 14:52:13.106467 9292 tablet_bootstrap.cc:654] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:13.106439 9293 consensus_queue.cc:260] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.107712 9294 tablet_bootstrap.cc:492] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: No bootstrap required, opened a new log
I20251027 14:52:13.107780 9293 ts_tablet_manager.cc:1434] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20251027 14:52:13.107856 9294 ts_tablet_manager.cc:1403] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Time spent bootstrapping tablet: real 0.004s user 0.003s sys 0.000s
I20251027 14:52:13.108611 9294 raft_consensus.cc:359] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.108873 9294 raft_consensus.cc:385] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:13.108984 9294 raft_consensus.cc:740] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 384c887b5f2246c9939e7b4156550f69, State: Initialized, Role: FOLLOWER
I20251027 14:52:13.109257 9294 consensus_queue.cc:260] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.109745 9294 ts_tablet_manager.cc:1434] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 14:52:13.110015 9292 tablet_bootstrap.cc:492] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: No bootstrap required, opened a new log
I20251027 14:52:13.110198 9292 ts_tablet_manager.cc:1403] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.001s
I20251027 14:52:13.110864 9292 raft_consensus.cc:359] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.111127 9292 raft_consensus.cc:385] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:13.111217 9292 raft_consensus.cc:740] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fb409e5bdffd43ffb50c754e647e7450, State: Initialized, Role: FOLLOWER
I20251027 14:52:13.111457 9292 consensus_queue.cc:260] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.111872 9292 ts_tablet_manager.cc:1434] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20251027 14:52:13.187904 9214 raft_consensus.cc:493] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:13.188158 9214 raft_consensus.cc:515] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.189733 9214 leader_election.cc:290] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:13.194941 9306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.190033 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:35615 (local address 127.0.104.2:33551)
1027 14:52:13.190723 (+ 690us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:13.190738 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:13.190862 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:13.191148 (+ 286us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:13.191152 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:13.191179 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:13.191715 (+ 536us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.191727 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.192978 (+ 1251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.192983 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:13.194272 (+ 1289us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.194290 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.194438 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.194464 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:52:13.194531 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:52:13.194647 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":495,"threads_started":1}
I20251027 14:52:13.195737 9305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.189920 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33363 (local address 127.0.104.2:44569)
1027 14:52:13.190655 (+ 735us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:13.190677 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:13.190828 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:13.191153 (+ 325us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:13.191158 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:13.191202 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:13.191864 (+ 662us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.191875 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.193154 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.193158 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:13.194327 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.194339 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.195337 (+ 998us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.195361 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:13.195425 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:52:13.195513 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":610,"spinlock_wait_cycles":12672,"thread_start_us":411,"threads_started":1}
I20251027 14:52:13.195937 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.190181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:33551 (local address 127.0.104.3:35615)
1027 14:52:13.190332 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.190338 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.190828 (+ 490us) server_negotiation.cc:408] Connection header received
1027 14:52:13.190986 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.190991 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.191046 (+ 55us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.191116 (+ 70us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:13.191854 (+ 738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.192844 (+ 990us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.194420 (+ 1576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.195430 (+ 1010us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.195481 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.195559 (+ 78us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.195688 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251027 14:52:13.195987 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.190160 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:44569 (local address 127.0.104.1:33363)
1027 14:52:13.190340 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.190347 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.190785 (+ 438us) server_negotiation.cc:408] Connection header received
1027 14:52:13.190964 (+ 179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.190969 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.191042 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.191119 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:13.192011 (+ 892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.192977 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.194507 (+ 1530us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.195560 (+ 1053us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.195603 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.195675 (+ 72us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.195785 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251027 14:52:13.196487 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "899c85be64064ef1b9aa16765148b9e3" candidate_uuid: "fb409e5bdffd43ffb50c754e647e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69" is_pre_election: true
I20251027 14:52:13.196643 8855 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "899c85be64064ef1b9aa16765148b9e3" candidate_uuid: "fb409e5bdffd43ffb50c754e647e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa892c685384a6195fe18e0ad41f885" is_pre_election: true
I20251027 14:52:13.196696 9139 raft_consensus.cc:2468] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb409e5bdffd43ffb50c754e647e7450 in term 0.
I20251027 14:52:13.196846 8855 raft_consensus.cc:2468] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate fb409e5bdffd43ffb50c754e647e7450 in term 0.
I20251027 14:52:13.197175 8925 leader_election.cc:304] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 384c887b5f2246c9939e7b4156550f69, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:13.197548 9214 raft_consensus.cc:2804] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:13.197659 9214 raft_consensus.cc:493] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:13.197739 9214 raft_consensus.cc:3060] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:13.199939 9214 raft_consensus.cc:515] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.200520 9214 leader_election.cc:290] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [CANDIDATE]: Term 1 election: Requested vote from peers 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363), 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615)
I20251027 14:52:13.200980 9139 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "899c85be64064ef1b9aa16765148b9e3" candidate_uuid: "fb409e5bdffd43ffb50c754e647e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "384c887b5f2246c9939e7b4156550f69"
I20251027 14:52:13.200992 8855 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "899c85be64064ef1b9aa16765148b9e3" candidate_uuid: "fb409e5bdffd43ffb50c754e647e7450" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "1fa892c685384a6195fe18e0ad41f885"
I20251027 14:52:13.201143 8855 raft_consensus.cc:3060] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:13.201166 9139 raft_consensus.cc:3060] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:13.203291 8855 raft_consensus.cc:2468] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb409e5bdffd43ffb50c754e647e7450 in term 1.
I20251027 14:52:13.203408 9139 raft_consensus.cc:2468] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate fb409e5bdffd43ffb50c754e647e7450 in term 1.
I20251027 14:52:13.203737 8924 leader_election.cc:304] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 1fa892c685384a6195fe18e0ad41f885, fb409e5bdffd43ffb50c754e647e7450; no voters:
I20251027 14:52:13.204015 9214 raft_consensus.cc:2804] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:13.204452 9214 raft_consensus.cc:697] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 1 LEADER]: Becoming Leader. State: Replica: fb409e5bdffd43ffb50c754e647e7450, State: Running, Role: LEADER
I20251027 14:52:13.204816 9214 consensus_queue.cc:237] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } }
I20251027 14:52:13.208212 8688 catalog_manager.cc:5649] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 reported cstate change: term changed from 0 to 1, leader changed from <none> to fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2). New cstate: current_term: 1 leader_uuid: "fb409e5bdffd43ffb50c754e647e7450" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fb409e5bdffd43ffb50c754e647e7450" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34545 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 } health_report { overall_health: UNKNOWN } } }
I20251027 14:52:13.234527 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.229628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45350 (local address 127.0.104.2:34545)
1027 14:52:13.230091 (+ 463us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.230095 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.230111 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:52:13.230188 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.230192 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.230266 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.230396 (+ 130us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:13.230902 (+ 506us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.231920 (+ 1018us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.233005 (+ 1085us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.233326 (+ 321us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.233399 (+ 73us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:13.234010 (+ 611us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:13.234124 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.234225 (+ 101us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.234299 (+ 74us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":360,"thread_start_us":264,"threads_started":1}
I20251027 14:52:13.248469 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.243728 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:34741 (local address 127.0.104.3:35615)
1027 14:52:13.243866 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.243871 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.244421 (+ 550us) server_negotiation.cc:408] Connection header received
1027 14:52:13.244641 (+ 220us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.244645 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.244726 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.244799 (+ 73us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:13.245407 (+ 608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.246194 (+ 787us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.247330 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.247716 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.247771 (+ 55us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:13.247887 (+ 116us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:13.247963 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.248083 (+ 120us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.248174 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:52:13.248517 9312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.243748 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:35615 (local address 127.0.104.2:34741)
1027 14:52:13.244313 (+ 565us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:13.244329 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:13.244506 (+ 177us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:13.244813 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:13.244816 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:13.244826 (+ 10us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:52:13.245293 (+ 467us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.245301 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.246342 (+ 1041us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.246346 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:13.247195 (+ 849us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.247207 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.247365 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.247431 (+ 66us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:52:13.247984 (+ 553us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:52:13.247995 (+ 11us) client_negotiation.cc:769] Sending connection context
1027 14:52:13.248059 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:52:13.248179 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":399,"thread_start_us":258,"threads_started":1}
I20251027 14:52:13.256072 9139 raft_consensus.cc:1275] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Refusing update from remote peer fb409e5bdffd43ffb50c754e647e7450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:13.256057 8855 raft_consensus.cc:1275] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Refusing update from remote peer fb409e5bdffd43ffb50c754e647e7450: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:13.256690 9307 consensus_queue.cc:1048] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "1fa892c685384a6195fe18e0ad41f885" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33363 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:13.256932 9214 consensus_queue.cc:1048] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [LEADER]: Connected to new peer: Peer: permanent_uuid: "384c887b5f2246c9939e7b4156550f69" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 35615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:13.287272 9317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.282555 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:34545 (local address 127.0.104.3:48621)
1027 14:52:13.282973 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:13.282988 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:13.283087 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:13.283391 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:13.283394 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:13.283402 (+ 8us) client_negotiation.cc:189] Negotiated authn=TOKEN
1027 14:52:13.283880 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.283891 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.285171 (+ 1280us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.285175 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:13.285966 (+ 791us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:13.285978 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.286107 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.286151 (+ 44us) client_negotiation.cc:252] Sending TOKEN_EXCHANGE NegotiatePB request
1027 14:52:13.286757 (+ 606us) client_negotiation.cc:271] Received TOKEN_EXCHANGE NegotiatePB response
1027 14:52:13.286768 (+ 11us) client_negotiation.cc:769] Sending connection context
1027 14:52:13.286839 (+ 71us) client_negotiation.cc:240] Negotiation successful
1027 14:52:13.286969 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":292,"thread_start_us":214,"threads_started":1}
I20251027 14:52:13.287273 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.282632 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:48621 (local address 127.0.104.2:34545)
1027 14:52:13.282770 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.282774 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.283057 (+ 283us) server_negotiation.cc:408] Connection header received
1027 14:52:13.283208 (+ 151us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.283213 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.283276 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.283380 (+ 104us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:13.284017 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.285041 (+ 1024us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.286095 (+ 1054us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.286467 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.286523 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:13.286656 (+ 133us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:13.286732 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.286884 (+ 152us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.286982 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:52:13.318989 8688 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40024:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:13.319228 8688 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:40024:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:13.322599 8688 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 09e138be90f842f098cec4a16f28e5ed: Sending DeleteTablet for 3 replicas of tablet 899c85be64064ef1b9aa16765148b9e3
I20251027 14:52:13.323406 8971 tablet_service.cc:1552] Processing DeleteTablet for tablet 899c85be64064ef1b9aa16765148b9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45340
I20251027 14:52:13.323494 8829 tablet_service.cc:1552] Processing DeleteTablet for tablet 899c85be64064ef1b9aa16765148b9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40188
I20251027 14:52:13.323735 9277 tablet_replica.cc:333] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: stopping tablet replica
I20251027 14:52:13.323783 9276 tablet_replica.cc:333] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: stopping tablet replica
I20251027 14:52:13.323930 9277 raft_consensus.cc:2243] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:13.323942 9276 raft_consensus.cc:2243] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:13.324164 9276 raft_consensus.cc:2272] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:13.324515 9277 raft_consensus.cc:2272] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:13.325635 9276 ts_tablet_manager.cc:1916] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:13.325651 9277 ts_tablet_manager.cc:1916] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:13.332913 9276 ts_tablet_manager.cc:1929] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:52:13.333145 9276 log.cc:1199] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/899c85be64064ef1b9aa16765148b9e3
I20251027 14:52:13.333195 9277 ts_tablet_manager.cc:1929] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:52:13.333350 9277 log.cc:1199] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/899c85be64064ef1b9aa16765148b9e3
I20251027 14:52:13.333779 9277 ts_tablet_manager.cc:1950] T 899c85be64064ef1b9aa16765148b9e3 P fb409e5bdffd43ffb50c754e647e7450: Deleting consensus metadata
I20251027 14:52:13.333783 9276 ts_tablet_manager.cc:1950] T 899c85be64064ef1b9aa16765148b9e3 P 1fa892c685384a6195fe18e0ad41f885: Deleting consensus metadata
I20251027 14:52:13.334815 9114 tablet_service.cc:1552] Processing DeleteTablet for tablet 899c85be64064ef1b9aa16765148b9e3 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:13 UTC) from {username='oryx', principal='oryx/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:42310
I20251027 14:52:13.335129 9278 tablet_replica.cc:333] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: stopping tablet replica
I20251027 14:52:13.335323 9278 raft_consensus.cc:2243] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:13.335500 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 8764
I20251027 14:52:13.335530 9278 raft_consensus.cc:2272] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:13.336038 8676 catalog_manager.cc:4985] TS fb409e5bdffd43ffb50c754e647e7450 (127.0.104.2:34545): tablet 899c85be64064ef1b9aa16765148b9e3 (table test-table [id=a9c386befd614377ba42b28258073e87]) successfully deleted
I20251027 14:52:13.336854 9278 ts_tablet_manager.cc:1916] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:13.344250 9278 ts_tablet_manager.cc:1929] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.5
I20251027 14:52:13.344614 9278 log.cc:1199] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonDefaultPrincipal.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/899c85be64064ef1b9aa16765148b9e3
I20251027 14:52:13.345090 9278 ts_tablet_manager.cc:1950] T 899c85be64064ef1b9aa16765148b9e3 P 384c887b5f2246c9939e7b4156550f69: Deleting consensus metadata
I20251027 14:52:13.345419 8677 catalog_manager.cc:4985] TS 1fa892c685384a6195fe18e0ad41f885 (127.0.104.1:33363): tablet 899c85be64064ef1b9aa16765148b9e3 (table test-table [id=a9c386befd614377ba42b28258073e87]) successfully deleted
I20251027 14:52:13.346237 8678 catalog_manager.cc:4985] TS 384c887b5f2246c9939e7b4156550f69 (127.0.104.3:35615): tablet 899c85be64064ef1b9aa16765148b9e3 (table test-table [id=a9c386befd614377ba42b28258073e87]) successfully deleted
Oct 27 14:52:13 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.1@KRBTEST.COM
I20251027 14:52:13.359346 9203 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.337084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40210 (local address 127.0.104.1:33363)
1027 14:52:13.337238 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.337244 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.346644 (+ 9400us) server_negotiation.cc:408] Connection header received
1027 14:52:13.346881 (+ 237us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.346886 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.346955 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.347055 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:13.348449 (+ 1394us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.349260 (+ 811us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.350407 (+ 1147us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.350872 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.353951 (+ 3079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:13.353999 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:13.354014 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:13.354044 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:13.356824 (+ 2780us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:13.357429 (+ 605us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:13.357437 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:13.357439 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:13.357520 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:13.357796 (+ 276us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:13.357801 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:13.357804 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:13.358076 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:13.358333 (+ 257us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.358755 (+ 422us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.359010 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:52:13.359980 8895 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:13.484216 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 8764
I20251027 14:52:13.504045 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 8906
Oct 27 14:52:13 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.2@KRBTEST.COM
I20251027 14:52:13.518564 9309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.504860 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45366 (local address 127.0.104.2:34545)
1027 14:52:13.505054 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.505059 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.505081 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:52:13.505214 (+ 133us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.505221 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.505284 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.505363 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:13.506675 (+ 1312us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.507675 (+ 1000us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.509005 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.509438 (+ 433us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.512694 (+ 3256us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:13.512728 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:13.512744 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:13.512775 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:13.515915 (+ 3140us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:13.516638 (+ 723us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:13.516648 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:13.516658 (+ 10us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:13.516762 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:13.517057 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:13.517062 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:13.517066 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:13.517486 (+ 420us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:13.517803 (+ 317us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.517937 (+ 134us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.518167 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20251027 14:52:13.519222 9037 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:13.641223 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 8906
I20251027 14:52:13.657513 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9048
Oct 27 14:52:13 dist-test-slave-hxxt krb5kdc[8643](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576732, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for oryx/127.0.104.3@KRBTEST.COM
I20251027 14:52:13.670585 9256 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:13.658153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42316 (local address 127.0.104.3:35615)
1027 14:52:13.658302 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:13.658308 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:13.658353 (+ 45us) server_negotiation.cc:408] Connection header received
1027 14:52:13.658519 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:13.658523 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:13.658590 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:13.658680 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:13.660050 (+ 1370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.660940 (+ 890us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:13.662039 (+ 1099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:13.662406 (+ 367us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:13.665705 (+ 3299us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:13.665743 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:13.665759 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:13.665793 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:13.668674 (+ 2881us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:13.669245 (+ 571us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:13.669252 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:13.669254 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:13.669320 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:13.669534 (+ 214us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:13.669536 (+ 2us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:13.669538 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:13.669795 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:13.670007 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:13.670171 (+ 164us) server_negotiation.cc:300] Negotiation successful
1027 14:52:13.670336 (+ 165us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:52:13.671205 9179 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:13.791427 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9048
I20251027 14:52:13.808488 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 8659
I20251027 14:52:13.809582 8731 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:13.922058 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 8659
2025-10-27T14:52:13Z chronyd exiting
[ OK ] SecurityITest.TestNonDefaultPrincipal (6337 ms)
[ RUN ] SecurityITest.TestNonExistentPrincipal
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:52:14 dist-test-slave-hxxt krb5kdc[9326](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:52:14 dist-test-slave-hxxt krb5kdc[9326](info): set up 2 sockets
Oct 27 14:52:14 dist-test-slave-hxxt krb5kdc[9326](info): commencing operation
krb5kdc: starting...
W20251027 14:52:15.796904 416 mini_kdc.cc:121] Time spent starting KDC: real 1.835s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:52:15 dist-test-slave-hxxt krb5kdc[9326](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576735, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:52:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:16.051095 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40739
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42355
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40739
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--principal=oryx with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:16.373728 9342 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:16.374051 9342 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:16.374190 9342 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:16.382112 9342 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:16.382229 9342 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:16.382310 9342 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:16.382370 9342 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:16.382452 9342 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:16.392657 9342 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42355
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40739
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=oryx
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40739
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestNonExistentPrincipal.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:16.394186 9342 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:16.395488 9342 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:16.404220 9347 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:16.404592 9348 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:16.404465 9350 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:16.405649 9342 server_base.cc:1047] running on GCE node
I20251027 14:52:16.406435 9342 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:16.408243 9342 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:16.409497 9342 hybrid_clock.cc:648] HybridClock initialized: now 1761576736409419 us; error 51 us; skew 500 ppm
I20251027 14:52:16.409821 9342 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20251027 14:52:16.413606 9342 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Keytab contains no suitable keys for oryx@KRBTEST.COM
2025-10-27T14:52:16Z chronyd exiting
[ OK ] SecurityITest.TestNonExistentPrincipal (2551 ms)
[ RUN ] SecurityITest.TestMismatchingPrincipals
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:52:16 dist-test-slave-hxxt krb5kdc[9356](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:52:16 dist-test-slave-hxxt krb5kdc[9356](info): set up 2 sockets
Oct 27 14:52:16 dist-test-slave-hxxt krb5kdc[9356](info): commencing operation
krb5kdc: starting...
W20251027 14:52:18.356900 416 mini_kdc.cc:121] Time spent starting KDC: real 1.844s user 0.001s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:52:18 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:52:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:18.605958 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41623
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:18.927910 9372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:18.928263 9372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:18.928422 9372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:18.936421 9372 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:18.936538 9372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:18.936618 9372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:18.936681 9372 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:18.936761 9372 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:18.947220 9372 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:18.948836 9372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:18.950155 9372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:18.957872 9378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:18.957860 9377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:18.958552 9380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:18.959053 9372 server_base.cc:1047] running on GCE node
I20251027 14:52:18.959780 9372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:18.961192 9372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:18.962431 9372 hybrid_clock.cc:648] HybridClock initialized: now 1761576738962362 us; error 39 us; skew 500 ppm
I20251027 14:52:18.962755 9372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:18 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576738, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:18.965782 9372 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:52:18.967344 9372 webserver.cc:492] Webserver started at http://127.0.104.62:42883/ using document root <none> and password file <none>
I20251027 14:52:18.968063 9372 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:18.968140 9372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:18.968513 9372 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:18.971058 9372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "33dea910a70240ee8764b750112cc5bd"
format_stamp: "Formatted at 2025-10-27 14:52:18 on dist-test-slave-hxxt"
server_key: "41499a0c6c36547b80c7f01a3196498f"
server_key_iv: "ea39ff941c8db11e13854f939a5c7914"
server_key_version: "encryptionkey@0"
I20251027 14:52:18.971791 9372 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "33dea910a70240ee8764b750112cc5bd"
format_stamp: "Formatted at 2025-10-27 14:52:18 on dist-test-slave-hxxt"
server_key: "41499a0c6c36547b80c7f01a3196498f"
server_key_iv: "ea39ff941c8db11e13854f939a5c7914"
server_key_version: "encryptionkey@0"
I20251027 14:52:18.977044 9372 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251027 14:52:18.980321 9387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:18.981638 9372 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:18.981788 9372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "33dea910a70240ee8764b750112cc5bd"
format_stamp: "Formatted at 2025-10-27 14:52:18 on dist-test-slave-hxxt"
server_key: "41499a0c6c36547b80c7f01a3196498f"
server_key_iv: "ea39ff941c8db11e13854f939a5c7914"
server_key_version: "encryptionkey@0"
I20251027 14:52:18.982002 9372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:18.999382 9372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:19.003396 9372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:19.003666 9372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:19.022599 9372 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41623
I20251027 14:52:19.022616 9438 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41623 every 8 connection(s)
I20251027 14:52:19.024024 9372 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:52:19.026654 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9372
I20251027 14:52:19.026866 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:52:19.027426 416 external_mini_cluster.cc:1468] Setting key 6b63b026461c7e51aaedda301bbc63a5
I20251027 14:52:19.028311 9439 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 27 14:52:19 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:19.040984 9439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Bootstrap starting.
I20251027 14:52:19.045122 9439 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:19.046452 9439 log.cc:826] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:19.048988 9442 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:19.029421 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60252 (local address 127.0.104.62:41623)
1027 14:52:19.030162 (+ 741us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:19.030177 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:19.030226 (+ 49us) server_negotiation.cc:408] Connection header received
1027 14:52:19.031675 (+ 1449us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:19.031706 (+ 31us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:19.032258 (+ 552us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:19.032789 (+ 531us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:19.034293 (+ 1504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:19.036283 (+ 1990us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:19.037453 (+ 1170us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:19.038030 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:19.040896 (+ 2866us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:19.041009 (+ 113us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:19.041030 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:19.041072 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:19.044704 (+ 3632us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:19.045423 (+ 719us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:19.045436 (+ 13us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:19.045446 (+ 10us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:19.045580 (+ 134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:19.045944 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:19.045949 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:19.045953 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:19.046592 (+ 639us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:19.046905 (+ 313us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:19.047397 (+ 492us) server_negotiation.cc:300] Negotiation successful
1027 14:52:19.047955 (+ 558us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":528,"thread_start_us":337,"threads_started":1}
I20251027 14:52:19.049646 9439 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: No bootstrap required, opened a new log
I20251027 14:52:19.055159 9439 raft_consensus.cc:359] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:19.055485 9439 raft_consensus.cc:385] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:19.055576 9439 raft_consensus.cc:740] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dea910a70240ee8764b750112cc5bd, State: Initialized, Role: FOLLOWER
I20251027 14:52:19.056154 9439 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:19.056346 9439 raft_consensus.cc:399] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:52:19.056521 9439 raft_consensus.cc:493] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:52:19.056669 9439 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:19.059207 9439 raft_consensus.cc:515] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:19.059680 9439 leader_election.cc:304] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33dea910a70240ee8764b750112cc5bd; no voters:
I20251027 14:52:19.060242 9439 leader_election.cc:290] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:52:19.060364 9444 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:19.060969 9444 raft_consensus.cc:697] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 LEADER]: Becoming Leader. State: Replica: 33dea910a70240ee8764b750112cc5bd, State: Running, Role: LEADER
I20251027 14:52:19.061465 9444 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:19.062171 9439 sys_catalog.cc:565] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:52:19.064175 9446 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33dea910a70240ee8764b750112cc5bd. Latest consensus state: current_term: 1 leader_uuid: "33dea910a70240ee8764b750112cc5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } } }
I20251027 14:52:19.064509 9446 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:19.065253 9445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "33dea910a70240ee8764b750112cc5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } } }
I20251027 14:52:19.065541 9445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:19.066082 9450 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:19.071748 9450 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:19.080394 9450 catalog_manager.cc:1357] Generated new cluster ID: e6c01d2c8c024861a86e239abd7bbb22
I20251027 14:52:19.080513 9450 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:19.119174 9450 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:52:19.120405 9450 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:19.130501 9450 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Generated new TSK 0
I20251027 14:52:19.131327 9450 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:19.232537 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41623
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:19.553429 9467 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:19.553755 9467 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:19.553891 9467 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:19.562127 9467 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:19.562237 9467 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:19.562422 9467 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:19.572816 9467 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:19.574373 9467 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:19.575618 9467 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:19.584758 9472 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:19.584764 9473 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:19.585250 9475 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:19.585634 9467 server_base.cc:1047] running on GCE node
I20251027 14:52:19.586218 9467 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:19.587168 9467 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:19.588446 9467 hybrid_clock.cc:648] HybridClock initialized: now 1761576739588353 us; error 43 us; skew 500 ppm
I20251027 14:52:19.588732 9467 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:19 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:19.591745 9467 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:52:19.593380 9467 webserver.cc:492] Webserver started at http://127.0.104.1:38487/ using document root <none> and password file <none>
I20251027 14:52:19.594162 9467 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:19.594255 9467 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:19.594610 9467 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:19.597314 9467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "7962fb678a2242e5acc8200983e15f92"
format_stamp: "Formatted at 2025-10-27 14:52:19 on dist-test-slave-hxxt"
server_key: "0823abf14593c02870f95f81a7c65fba"
server_key_iv: "3b150aace4e0ad26788a20b1e41ece60"
server_key_version: "encryptionkey@0"
I20251027 14:52:19.598094 9467 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "7962fb678a2242e5acc8200983e15f92"
format_stamp: "Formatted at 2025-10-27 14:52:19 on dist-test-slave-hxxt"
server_key: "0823abf14593c02870f95f81a7c65fba"
server_key_iv: "3b150aace4e0ad26788a20b1e41ece60"
server_key_version: "encryptionkey@0"
I20251027 14:52:19.603492 9467 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20251027 14:52:19.606926 9482 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:19.608284 9467 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:52:19.608464 9467 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "7962fb678a2242e5acc8200983e15f92"
format_stamp: "Formatted at 2025-10-27 14:52:19 on dist-test-slave-hxxt"
server_key: "0823abf14593c02870f95f81a7c65fba"
server_key_iv: "3b150aace4e0ad26788a20b1e41ece60"
server_key_version: "encryptionkey@0"
I20251027 14:52:19.608676 9467 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:19.619657 9467 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:19.623502 9467 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:19.623759 9467 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:19.624780 9467 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:19.626206 9467 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:19.626302 9467 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:19.626400 9467 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:19.626471 9467 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:19.656020 9467 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:44087
I20251027 14:52:19.656085 9594 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:44087 every 8 connection(s)
I20251027 14:52:19.657392 9467 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:19.665584 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9467
I20251027 14:52:19.665779 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:52:19.666496 416 external_mini_cluster.cc:1468] Setting key 220981db6fb9ea025ad375ab8dec7590
Oct 27 14:52:19 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576739, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:19.675297 9597 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:19.659964 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:42771 (local address 127.0.104.62:41623)
1027 14:52:19.660406 (+ 442us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:19.660412 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:19.661550 (+ 1138us) server_negotiation.cc:408] Connection header received
1027 14:52:19.663064 (+ 1514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:19.663070 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:19.663151 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:19.663251 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:19.665024 (+ 1773us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:19.665963 (+ 939us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:19.667071 (+ 1108us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:19.667536 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:19.670298 (+ 2762us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:19.670326 (+ 28us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:19.670329 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:19.670357 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:19.672623 (+ 2266us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:19.673275 (+ 652us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:19.673280 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:19.673282 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:19.673366 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:19.673666 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:19.673670 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:19.673672 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:19.673894 (+ 222us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:19.674016 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:19.674718 (+ 702us) server_negotiation.cc:300] Negotiation successful
1027 14:52:19.674967 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":220,"threads_started":1}
I20251027 14:52:19.676190 9598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:19.660628 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:42771)
1027 14:52:19.661385 (+ 757us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:19.661424 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:19.662733 (+ 1309us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:19.663507 (+ 774us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:19.663522 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:19.664109 (+ 587us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:19.664831 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:19.664850 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:19.666107 (+ 1257us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:19.666113 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:19.666939 (+ 826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:19.666950 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:19.667198 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:19.667989 (+ 791us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:19.668013 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:19.670121 (+ 2108us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:19.672746 (+ 2625us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:19.672755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:19.672773 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:19.673160 (+ 387us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:19.673461 (+ 301us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:19.673465 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:19.673467 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:19.673566 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:19.674006 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:19.674014 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:19.674326 (+ 312us) client_negotiation.cc:769] Sending connection context
1027 14:52:19.674610 (+ 284us) client_negotiation.cc:240] Negotiation successful
1027 14:52:19.675070 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":524,"thread_start_us":362,"threads_started":1}
I20251027 14:52:19.677750 9595 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41623
I20251027 14:52:19.678105 9595 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:19.678856 9595 heartbeater.cc:507] Master 127.0.104.62:41623 requested a full tablet report, sending...
I20251027 14:52:19.680915 9404 ts_manager.cc:194] Registered new tserver with Master: 7962fb678a2242e5acc8200983e15f92 (127.0.104.1:44087)
I20251027 14:52:19.682353 9404 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:42771
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:19.752188 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41623
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:20.077379 9603 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:20.077680 9603 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:20.077783 9603 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:20.085842 9603 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:20.085951 9603 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:20.086154 9603 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:20.096495 9603 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:20.098104 9603 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:20.099392 9603 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:20.108556 9609 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:20.108548 9608 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:20.109400 9611 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:20.109802 9603 server_base.cc:1047] running on GCE node
I20251027 14:52:20.110378 9603 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:20.111246 9603 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:20.112466 9603 hybrid_clock.cc:648] HybridClock initialized: now 1761576740112399 us; error 38 us; skew 500 ppm
I20251027 14:52:20.112741 9603 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:20 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:20.115826 9603 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:52:20.117426 9603 webserver.cc:492] Webserver started at http://127.0.104.2:44437/ using document root <none> and password file <none>
I20251027 14:52:20.118152 9603 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:20.118243 9603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:20.118587 9603 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:20.121227 9603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "59622de35e5e4a1ca08416a6858a08c9"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "709eb7be6f1996ab060ec52bcb17a555"
server_key_iv: "95a9a1876d876857be9a407e41467994"
server_key_version: "encryptionkey@0"
I20251027 14:52:20.121978 9603 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "59622de35e5e4a1ca08416a6858a08c9"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "709eb7be6f1996ab060ec52bcb17a555"
server_key_iv: "95a9a1876d876857be9a407e41467994"
server_key_version: "encryptionkey@0"
I20251027 14:52:20.127343 9603 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:52:20.130698 9618 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:20.132032 9603 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:52:20.132194 9603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "59622de35e5e4a1ca08416a6858a08c9"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "709eb7be6f1996ab060ec52bcb17a555"
server_key_iv: "95a9a1876d876857be9a407e41467994"
server_key_version: "encryptionkey@0"
I20251027 14:52:20.132417 9603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:20.149122 9603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:20.153316 9603 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:20.153582 9603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:20.154556 9603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:20.156068 9603 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:20.156163 9603 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:20.156266 9603 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:20.156338 9603 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:20.188019 9603 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:39973
I20251027 14:52:20.188077 9730 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:39973 every 8 connection(s)
I20251027 14:52:20.189414 9603 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:20.192718 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9603
I20251027 14:52:20.192905 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:52:20.193399 416 external_mini_cluster.cc:1468] Setting key 5ab49d944533bc812c24ef01e13d8f7f
Oct 27 14:52:20 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:20.209709 9733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:20.192040 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:54867 (local address 127.0.104.62:41623)
1027 14:52:20.192470 (+ 430us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:20.192476 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:20.193594 (+ 1118us) server_negotiation.cc:408] Connection header received
1027 14:52:20.195375 (+ 1781us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:20.195379 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:20.195444 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:20.195546 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:20.197759 (+ 2213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.198668 (+ 909us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:20.200031 (+ 1363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.200337 (+ 306us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:20.203493 (+ 3156us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:20.203547 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:20.203553 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:20.203590 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:20.206771 (+ 3181us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:20.207506 (+ 735us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:20.207512 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:20.207515 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:20.207626 (+ 111us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:20.207924 (+ 298us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:20.207928 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:20.207932 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:20.208217 (+ 285us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:20.208349 (+ 132us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:20.209156 (+ 807us) server_negotiation.cc:300] Negotiation successful
1027 14:52:20.209393 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":289,"thread_start_us":186,"threads_started":1}
I20251027 14:52:20.210564 9734 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:20.192617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:54867)
1027 14:52:20.193418 (+ 801us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:20.193467 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:20.195069 (+ 1602us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:20.195853 (+ 784us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:20.195866 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:20.196620 (+ 754us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:20.197587 (+ 967us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:20.197614 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.198826 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:20.198831 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:20.199890 (+ 1059us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:20.199903 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.200157 (+ 254us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:20.200896 (+ 739us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:20.200926 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:20.203300 (+ 2374us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:20.206947 (+ 3647us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:20.206955 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:20.206970 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:20.207384 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:20.207716 (+ 332us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:20.207719 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:20.207722 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:20.207836 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:20.208363 (+ 527us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:20.208371 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:20.208745 (+ 374us) client_negotiation.cc:769] Sending connection context
1027 14:52:20.209041 (+ 296us) client_negotiation.cc:240] Negotiation successful
1027 14:52:20.209435 (+ 394us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":515,"thread_start_us":347,"threads_started":1}
I20251027 14:52:20.211988 9731 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41623
I20251027 14:52:20.212325 9731 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:20.213224 9731 heartbeater.cc:507] Master 127.0.104.62:41623 requested a full tablet report, sending...
I20251027 14:52:20.214828 9404 ts_manager.cc:194] Registered new tserver with Master: 59622de35e5e4a1ca08416a6858a08c9 (127.0.104.2:39973)
I20251027 14:52:20.215924 9404 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:54867
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:20.287204 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41623
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:20.610105 9739 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:20.610419 9739 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:20.610555 9739 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:20.618744 9739 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:20.618861 9739 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:20.619033 9739 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:20.629479 9739 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:20.631045 9739 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:20.632409 9739 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:20.641683 9745 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:20.641659 9744 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:20.642747 9747 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:20.642834 9739 server_base.cc:1047] running on GCE node
I20251027 14:52:20.643435 9739 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:20.644325 9739 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:20.645612 9739 hybrid_clock.cc:648] HybridClock initialized: now 1761576740645535 us; error 42 us; skew 500 ppm
I20251027 14:52:20.645956 9739 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:20 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:20.649180 9739 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:52:20.650743 9739 webserver.cc:492] Webserver started at http://127.0.104.3:44425/ using document root <none> and password file <none>
I20251027 14:52:20.651440 9739 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:20.651526 9739 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:20.651917 9739 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:20.654482 9739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "ded8ae81ede941be8d4ddf1c129a545b"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "74183d20354a29ddd7058b8b50199467"
server_key_iv: "c209eba5c7e31c88bce62065d474db90"
server_key_version: "encryptionkey@0"
I20251027 14:52:20.655251 9739 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "ded8ae81ede941be8d4ddf1c129a545b"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "74183d20354a29ddd7058b8b50199467"
server_key_iv: "c209eba5c7e31c88bce62065d474db90"
server_key_version: "encryptionkey@0"
I20251027 14:52:20.660633 9739 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20251027 14:52:20.664026 9754 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:20.665328 9739 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:20.665481 9739 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "ded8ae81ede941be8d4ddf1c129a545b"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "74183d20354a29ddd7058b8b50199467"
server_key_iv: "c209eba5c7e31c88bce62065d474db90"
server_key_version: "encryptionkey@0"
I20251027 14:52:20.665673 9739 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:20.685191 9595 heartbeater.cc:499] Master 127.0.104.62:41623 was elected leader, sending a full tablet report...
I20251027 14:52:20.692095 9739 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:20.695971 9739 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:20.696219 9739 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:20.697206 9739 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:20.698621 9739 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:20.698715 9739 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:20.698828 9739 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:20.698895 9739 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:20.728183 9739 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36429
I20251027 14:52:20.728227 9866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36429 every 8 connection(s)
I20251027 14:52:20.729530 9739 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:20.737793 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9739
I20251027 14:52:20.737996 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:52:20.738461 416 external_mini_cluster.cc:1468] Setting key 5e32170a1f6003f7fd2fa1a17a33be4d
Oct 27 14:52:20 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576740, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:20.747714 9869 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:20.732107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:47691 (local address 127.0.104.62:41623)
1027 14:52:20.732541 (+ 434us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:20.732545 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:20.733517 (+ 972us) server_negotiation.cc:408] Connection header received
1027 14:52:20.734974 (+ 1457us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:20.734980 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:20.735071 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:20.735169 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:20.736907 (+ 1738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.737814 (+ 907us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:20.739212 (+ 1398us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.739573 (+ 361us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:20.742651 (+ 3078us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:20.742682 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:20.742686 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:20.742716 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:20.745005 (+ 2289us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:20.745635 (+ 630us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:20.745640 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:20.745643 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:20.745713 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:20.746144 (+ 431us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:20.746147 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:20.746149 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:20.746369 (+ 220us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:20.746470 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:20.747239 (+ 769us) server_negotiation.cc:300] Negotiation successful
1027 14:52:20.747442 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":204,"threads_started":1}
I20251027 14:52:20.748714 9870 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:20.732690 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:47691)
1027 14:52:20.733366 (+ 676us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:20.733404 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:20.734699 (+ 1295us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:20.735430 (+ 731us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:20.735439 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:20.735987 (+ 548us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:20.736686 (+ 699us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:20.736707 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.737988 (+ 1281us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:20.738001 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:20.739075 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:20.739087 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.739334 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:20.740044 (+ 710us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:20.740077 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:20.742471 (+ 2394us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:20.745122 (+ 2651us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:20.745128 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:20.745143 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:20.745516 (+ 373us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:20.745913 (+ 397us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:20.745918 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:20.745920 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:20.746049 (+ 129us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:20.746489 (+ 440us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:20.746496 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:20.746818 (+ 322us) client_negotiation.cc:769] Sending connection context
1027 14:52:20.747111 (+ 293us) client_negotiation.cc:240] Negotiation successful
1027 14:52:20.747471 (+ 360us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":292,"threads_started":1}
I20251027 14:52:20.750062 9867 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41623
I20251027 14:52:20.750380 9867 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:20.751130 9867 heartbeater.cc:507] Master 127.0.104.62:41623 requested a full tablet report, sending...
I20251027 14:52:20.752616 9404 ts_manager.cc:194] Registered new tserver with Master: ded8ae81ede941be8d4ddf1c129a545b (127.0.104.3:36429)
I20251027 14:52:20.753531 9404 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:47691
I20251027 14:52:20.754081 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
WARNING: no policy specified for oryx/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.1.keytab.
Entry for principal oryx/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.1.keytab.
WARNING: no policy specified for oryx/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.2.keytab.
Entry for principal oryx/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.2.keytab.
WARNING: no policy specified for oryx/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "oryx/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal oryx/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.3.keytab.
Entry for principal oryx/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.3.keytab.
I20251027 14:52:20.879470 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9467
Oct 27 14:52:20 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:52:20.893898 9878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:20.880535 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41652 (local address 127.0.104.1:44087)
1027 14:52:20.881075 (+ 540us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:20.881082 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:20.881111 (+ 29us) server_negotiation.cc:408] Connection header received
1027 14:52:20.881211 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:20.881218 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:20.881421 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:20.881579 (+ 158us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:20.883032 (+ 1453us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.884209 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:20.885330 (+ 1121us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.885718 (+ 388us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:20.888936 (+ 3218us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:20.888996 (+ 60us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:20.889012 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:20.889041 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:20.891610 (+ 2569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:20.892167 (+ 557us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:20.892175 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:20.892178 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:20.892253 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:20.892530 (+ 277us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:20.892534 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:20.892536 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:20.892816 (+ 280us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:20.893045 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:20.893310 (+ 265us) server_negotiation.cc:300] Negotiation successful
1027 14:52:20.893607 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":298,"threads_started":1}
I20251027 14:52:20.895138 9590 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:20.977506 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9467
I20251027 14:52:20.991132 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9603
Oct 27 14:52:20 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:52:21.005820 9880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:20.992021 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57322 (local address 127.0.104.2:39973)
1027 14:52:20.992537 (+ 516us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:20.992545 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:20.992570 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:52:20.992637 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:20.992644 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:20.992863 (+ 219us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:20.993045 (+ 182us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:20.994455 (+ 1410us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.995600 (+ 1145us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:20.996820 (+ 1220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:20.997246 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:21.000481 (+ 3235us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:21.000546 (+ 65us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:21.000563 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:21.000596 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:21.003273 (+ 2677us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:21.003912 (+ 639us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:21.003920 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:21.003923 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:21.004001 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:21.004300 (+ 299us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:21.004306 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:21.004309 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:21.004653 (+ 344us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:21.004956 (+ 303us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:21.005226 (+ 270us) server_negotiation.cc:300] Negotiation successful
1027 14:52:21.005499 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":376,"thread_start_us":240,"threads_started":1}
I20251027 14:52:21.007114 9726 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:21.090559 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9603
I20251027 14:52:21.103152 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9739
Oct 27 14:52:21 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576738, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:52:21.118294 9882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:21.104058 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38092 (local address 127.0.104.3:36429)
1027 14:52:21.104649 (+ 591us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:21.104656 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:21.104684 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:52:21.104773 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:21.104779 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:21.104982 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:21.105129 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:21.106503 (+ 1374us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:21.107745 (+ 1242us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:21.109053 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:21.109458 (+ 405us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:21.112936 (+ 3478us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:21.113004 (+ 68us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:21.113020 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:21.113054 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:21.115946 (+ 2892us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:21.116562 (+ 616us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:21.116570 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:21.116573 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:21.116655 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:21.116911 (+ 256us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:21.116915 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:21.116917 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:21.117186 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:21.117397 (+ 211us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:21.117730 (+ 333us) server_negotiation.cc:300] Negotiation successful
1027 14:52:21.117989 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":460,"thread_start_us":341,"threads_started":1}
I20251027 14:52:21.119549 9862 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:21.202634 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9739
I20251027 14:52:21.215687 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9372
I20251027 14:52:21.216930 9434 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:21.317513 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9372
I20251027 14:52:21.334576 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41623
--webserver_interface=127.0.104.62
--webserver_port=42883
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41623
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:21.652186 9885 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:21.652549 9885 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:21.652684 9885 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:21.660707 9885 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:21.660822 9885 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:21.660902 9885 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:21.660971 9885 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:21.661051 9885 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:21.671434 9885 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41623
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41623
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=42883
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:21.673043 9885 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:21.674387 9885 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:21.682860 9890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:21.683079 9891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:21.682905 9893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:21.683539 9885 server_base.cc:1047] running on GCE node
I20251027 14:52:21.684309 9885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:21.685719 9885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:21.686951 9885 hybrid_clock.cc:648] HybridClock initialized: now 1761576741686876 us; error 45 us; skew 500 ppm
I20251027 14:52:21.687287 9885 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:21 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576741, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:21.690402 9885 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:52:21.691990 9885 webserver.cc:492] Webserver started at http://127.0.104.62:42883/ using document root <none> and password file <none>
I20251027 14:52:21.692787 9885 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:21.692891 9885 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:21.698637 9885 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251027 14:52:21.701702 9900 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:21.703213 9885 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:21.703399 9885 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "33dea910a70240ee8764b750112cc5bd"
format_stamp: "Formatted at 2025-10-27 14:52:18 on dist-test-slave-hxxt"
server_key: "41499a0c6c36547b80c7f01a3196498f"
server_key_iv: "ea39ff941c8db11e13854f939a5c7914"
server_key_version: "encryptionkey@0"
I20251027 14:52:21.704172 9885 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:21.750605 9885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:21.754384 9885 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:21.754675 9885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:21.772665 9885 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41623
I20251027 14:52:21.772668 9951 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41623 every 8 connection(s)
I20251027 14:52:21.774252 9885 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:52:21.775354 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9885
I20251027 14:52:21.776295 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:44087
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:41623
--webserver_port=38487
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.1.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:52:21.783758 9952 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:52:21.787218 9952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Bootstrap starting.
I20251027 14:52:21.804570 9952 log.cc:826] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:21.813707 9952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:52:21.814167 9952 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Bootstrap complete.
I20251027 14:52:21.820544 9952 raft_consensus.cc:359] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:21.821197 9952 raft_consensus.cc:740] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33dea910a70240ee8764b750112cc5bd, State: Initialized, Role: FOLLOWER
I20251027 14:52:21.821872 9952 consensus_queue.cc:260] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:21.822077 9952 raft_consensus.cc:399] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:52:21.822186 9952 raft_consensus.cc:493] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:52:21.822343 9952 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 1 FOLLOWER]: Advancing to term 2
I20251027 14:52:21.825950 9952 raft_consensus.cc:515] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:21.826472 9952 leader_election.cc:304] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 33dea910a70240ee8764b750112cc5bd; no voters:
I20251027 14:52:21.827078 9952 leader_election.cc:290] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20251027 14:52:21.827211 9956 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 2 FOLLOWER]: Leader election won for term 2
I20251027 14:52:21.828322 9956 raft_consensus.cc:697] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [term 2 LEADER]: Becoming Leader. State: Replica: 33dea910a70240ee8764b750112cc5bd, State: Running, Role: LEADER
I20251027 14:52:21.828876 9956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } }
I20251027 14:52:21.829387 9952 sys_catalog.cc:565] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:52:21.831384 9957 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "33dea910a70240ee8764b750112cc5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } } }
I20251027 14:52:21.831707 9957 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:21.835402 9958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 33dea910a70240ee8764b750112cc5bd. Latest consensus state: current_term: 2 leader_uuid: "33dea910a70240ee8764b750112cc5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "33dea910a70240ee8764b750112cc5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41623 } } }
I20251027 14:52:21.835695 9958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:21.837519 9961 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:21.843955 9961 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:21.845511 9961 catalog_manager.cc:1269] Loaded cluster ID: e6c01d2c8c024861a86e239abd7bbb22
I20251027 14:52:21.845628 9961 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:21.850031 9961 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:21.851800 9961 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 33dea910a70240ee8764b750112cc5bd: Loaded TSK: 0
I20251027 14:52:21.853783 9961 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251027 14:52:22.146688 9954 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:22.146996 9954 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:22.147123 9954 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:22.155169 9954 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:22.155275 9954 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:22.155463 9954 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:22.165747 9954 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.1.keytab
--principal=oryx/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:44087
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=38487
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:22.167305 9954 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:22.168642 9954 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:22.178288 9979 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:22.178670 9978 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:22.179427 9981 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:22.180799 9954 server_base.cc:1047] running on GCE node
I20251027 14:52:22.181420 9954 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:22.182407 9954 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:22.183665 9954 hybrid_clock.cc:648] HybridClock initialized: now 1761576742183548 us; error 70 us; skew 500 ppm
I20251027 14:52:22.183965 9954 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576742, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:22.187218 9954 init.cc:377] Logged in from keytab as oryx/127.0.104.1@KRBTEST.COM (short username oryx)
I20251027 14:52:22.189023 9954 webserver.cc:492] Webserver started at http://127.0.104.1:38487/ using document root <none> and password file <none>
I20251027 14:52:22.189790 9954 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:22.189905 9954 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:22.195964 9954 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20251027 14:52:22.199189 9988 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:22.200888 9954 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251027 14:52:22.201071 9954 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "7962fb678a2242e5acc8200983e15f92"
format_stamp: "Formatted at 2025-10-27 14:52:19 on dist-test-slave-hxxt"
server_key: "0823abf14593c02870f95f81a7c65fba"
server_key_iv: "3b150aace4e0ad26788a20b1e41ece60"
server_key_version: "encryptionkey@0"
I20251027 14:52:22.201798 9954 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:22.216078 9954 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:22.220041 9954 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:22.220305 9954 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:22.221267 9954 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:22.222858 9954 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:22.222947 9954 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:22.223063 9954 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:22.223114 9954 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:22.256261 9954 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:44087
I20251027 14:52:22.256314 10100 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:44087 every 8 connection(s)
I20251027 14:52:22.257889 9954 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:22.261351 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 9954
I20251027 14:52:22.262115 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:39973
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:41623
--webserver_port=44437
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.2.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:22.278564 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:22.261174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:57165)
1027 14:52:22.264961 (+ 3787us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:22.265013 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:22.266447 (+ 1434us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:22.268223 (+ 1776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:22.268235 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:22.269049 (+ 814us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:22.269727 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.269747 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.272206 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.272212 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:22.273192 (+ 980us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.273205 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.273433 (+ 228us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.274130 (+ 697us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:22.274159 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:22.277610 (+ 3451us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":3511,"thread_start_us":3329,"threads_started":1}
W20251027 14:52:22.279246 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:22.261051 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:57165 (local address 127.0.104.62:41623)
1027 14:52:22.261847 (+ 796us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:22.261865 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:22.265178 (+ 3313us) server_negotiation.cc:408] Connection header received
1027 14:52:22.267083 (+ 1905us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:22.267103 (+ 20us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:22.267610 (+ 507us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:22.267992 (+ 382us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:22.270009 (+ 2017us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.271967 (+ 1958us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.273331 (+ 1364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.273822 (+ 491us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.278033 (+ 4211us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:57165: BlockingRecv error: recv got EOF from 127.0.104.1:57165 (error 108)
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":348,"threads_started":1}
W20251027 14:52:22.279317 10101 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:41623 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.104.62:41623: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:22.287012 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:22.279925 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:43805)
1027 14:52:22.280056 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:22.280073 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:22.280166 (+ 93us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:22.280572 (+ 406us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:22.280577 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:22.280888 (+ 311us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:22.281182 (+ 294us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.281192 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.282312 (+ 1120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.282316 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:22.283065 (+ 749us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.283075 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.283228 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.283922 (+ 694us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:22.283940 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:22.286815 (+ 2875us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":35}
W20251027 14:52:22.287309 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:22.280107 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:43805 (local address 127.0.104.62:41623)
1027 14:52:22.280268 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:22.280273 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:22.280289 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:52:22.280337 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:22.280341 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:22.280440 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:22.280520 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:22.281322 (+ 802us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.282171 (+ 849us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.283193 (+ 1022us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.283632 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.287087 (+ 3455us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:43805: BlockingRecv error: recv got EOF from 127.0.104.1:43805 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:22.295540 10104 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:22.287726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:35561)
1027 14:52:22.287898 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:22.287917 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:22.288036 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:22.288316 (+ 280us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:22.288321 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:22.288768 (+ 447us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:22.289193 (+ 425us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.289205 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.290319 (+ 1114us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.290324 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:22.291467 (+ 1143us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.291482 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.291648 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.292334 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:22.292357 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:22.295344 (+ 2987us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20251027 14:52:22.295845 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:22.287778 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:35561 (local address 127.0.104.62:41623)
1027 14:52:22.287923 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:22.287927 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:22.287968 (+ 41us) server_negotiation.cc:408] Connection header received
1027 14:52:22.288134 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:22.288138 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:22.288197 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:22.288276 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:22.289352 (+ 1076us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.290193 (+ 841us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.291604 (+ 1411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.291956 (+ 352us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.295618 (+ 3662us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:35561: BlockingRecv error: recv got EOF from 127.0.104.1:35561 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
W20251027 14:52:22.295921 10101 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251027 14:52:22.593518 10105 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:22.593833 10105 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:22.593992 10105 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:22.602083 10105 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:22.602191 10105 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:22.602381 10105 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:22.612780 10105 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.2.keytab
--principal=oryx/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:39973
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=44437
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:22.614347 10105 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:22.615665 10105 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:22.624823 10110 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:22.624881 10111 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:22.625847 10113 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:22.626237 10105 server_base.cc:1047] running on GCE node
I20251027 14:52:22.626858 10105 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:22.627810 10105 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:22.629019 10105 hybrid_clock.cc:648] HybridClock initialized: now 1761576742628958 us; error 36 us; skew 500 ppm
I20251027 14:52:22.629305 10105 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576742, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:22.632324 10105 init.cc:377] Logged in from keytab as oryx/127.0.104.2@KRBTEST.COM (short username oryx)
I20251027 14:52:22.633960 10105 webserver.cc:492] Webserver started at http://127.0.104.2:44437/ using document root <none> and password file <none>
I20251027 14:52:22.634704 10105 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:22.634807 10105 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:22.640306 10105 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:22.643097 10120 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:22.644564 10105 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:22.644770 10105 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "59622de35e5e4a1ca08416a6858a08c9"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "709eb7be6f1996ab060ec52bcb17a555"
server_key_iv: "95a9a1876d876857be9a407e41467994"
server_key_version: "encryptionkey@0"
I20251027 14:52:22.645462 10105 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:22.668751 10105 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:22.677059 10105 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:22.677353 10105 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:22.678342 10105 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:22.679891 10105 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:22.679968 10105 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:22.680085 10105 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:22.680135 10105 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:22.710304 10105 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:39973
I20251027 14:52:22.710336 10232 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:39973 every 8 connection(s)
I20251027 14:52:22.711711 10105 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:22.715507 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10105
I20251027 14:52:22.716210 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:36429
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:41623
--webserver_port=44425
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:38493
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--principal=oryx/127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.3.keytab with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
W20251027 14:52:22.726754 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:22.714371 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:49087 (local address 127.0.104.62:41623)
1027 14:52:22.714528 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:22.714533 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:22.716096 (+ 1563us) server_negotiation.cc:408] Connection header received
1027 14:52:22.717699 (+ 1603us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:22.717706 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:22.717804 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:22.717899 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:22.719635 (+ 1736us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.720538 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.721913 (+ 1375us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.722339 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.726519 (+ 4180us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:49087: BlockingRecv error: recv got EOF from 127.0.104.2:49087 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:52:22.727301 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:22.714999 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:49087)
1027 14:52:22.715925 (+ 926us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:22.715964 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:22.717381 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:22.718143 (+ 762us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:22.718152 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:22.718779 (+ 627us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:22.719463 (+ 684us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.719482 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.720695 (+ 1213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.720701 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:22.721771 (+ 1070us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.721786 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.722049 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.722817 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:22.722863 (+ 46us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:22.726254 (+ 3391us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":640,"thread_start_us":382,"threads_started":1}
W20251027 14:52:22.727990 10233 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:41623 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.104.62:41623: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:22.735836 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:22.728592 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:49157)
1027 14:52:22.728812 (+ 220us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:22.728828 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:22.728971 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:22.729263 (+ 292us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:22.729268 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:22.729591 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:22.729907 (+ 316us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.729916 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.731265 (+ 1349us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.731269 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:22.732059 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.732071 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.732229 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.732907 (+ 678us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:22.732924 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:22.735594 (+ 2670us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":123}
W20251027 14:52:22.735975 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:22.728687 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:49157 (local address 127.0.104.62:41623)
1027 14:52:22.728838 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:22.728843 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:22.728916 (+ 73us) server_negotiation.cc:408] Connection header received
1027 14:52:22.729080 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:22.729084 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:22.729140 (+ 56us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:22.729226 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:22.730042 (+ 816us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.731127 (+ 1085us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.732190 (+ 1063us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.732625 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.735775 (+ 3150us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:49157: BlockingRecv error: recv got EOF from 127.0.104.2:49157 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:22 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:22.744055 10235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:22.736642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:49889)
1027 14:52:22.736777 (+ 135us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:22.736793 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:22.736903 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:22.737166 (+ 263us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:22.737171 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:22.737481 (+ 310us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:22.737766 (+ 285us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.737775 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.738899 (+ 1124us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.738905 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:22.740011 (+ 1106us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:22.740025 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.740165 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.740903 (+ 738us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:22.740927 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:22.743788 (+ 2861us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":42}
W20251027 14:52:22.744124 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:22.736770 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:49889 (local address 127.0.104.62:41623)
1027 14:52:22.736920 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:22.736925 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:22.736947 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:52:22.736993 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:22.736998 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:22.737057 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:22.737131 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:22.737902 (+ 771us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.738723 (+ 821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:22.740139 (+ 1416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:22.740474 (+ 335us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:22.743970 (+ 3496us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:49889: BlockingRecv error: recv got EOF from 127.0.104.2:49889 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
W20251027 14:52:22.744340 10233 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251027 14:52:23.038935 10236 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:23.039237 10236 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:23.039382 10236 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:23.047323 10236 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:23.047430 10236 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:23.047586 10236 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:23.058604 10236 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:38493
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/krb5kdc/oryx_127.0.104.3.keytab
--principal=oryx/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:36429
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=44425
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:41623
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:23.060102 10236 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:23.061422 10236 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:23.070456 10241 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:23.070451 10242 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:23.071079 10244 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:23.071441 10236 server_base.cc:1047] running on GCE node
I20251027 14:52:23.072048 10236 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:23.072973 10236 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:23.074225 10236 hybrid_clock.cc:648] HybridClock initialized: now 1761576743074145 us; error 40 us; skew 500 ppm
I20251027 14:52:23.074532 10236 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576743, etypes {rep=17 tkt=17 ses=17}, oryx/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:23.077533 10236 init.cc:377] Logged in from keytab as oryx/127.0.104.3@KRBTEST.COM (short username oryx)
I20251027 14:52:23.079078 10236 webserver.cc:492] Webserver started at http://127.0.104.3:44425/ using document root <none> and password file <none>
I20251027 14:52:23.079825 10236 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:23.079919 10236 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:23.085577 10236 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20251027 14:52:23.088498 10251 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:23.089867 10236 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:52:23.090025 10236 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "ded8ae81ede941be8d4ddf1c129a545b"
format_stamp: "Formatted at 2025-10-27 14:52:20 on dist-test-slave-hxxt"
server_key: "74183d20354a29ddd7058b8b50199467"
server_key_iv: "c209eba5c7e31c88bce62065d474db90"
server_key_version: "encryptionkey@0"
I20251027 14:52:23.090723 10236 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:23.104135 10236 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:23.108266 10236 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:23.108577 10236 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:23.109637 10236 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:23.111136 10236 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:23.111207 10236 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:23.111317 10236 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:23.111366 10236 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:23.140873 10236 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36429
I20251027 14:52:23.140938 10363 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36429 every 8 connection(s)
I20251027 14:52:23.142247 10236 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestMismatchingPrincipals.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:23.148401 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10236
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
W20251027 14:52:23.158224 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:23.144880 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:53361 (local address 127.0.104.62:41623)
1027 14:52:23.145060 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:23.145065 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:23.146366 (+ 1301us) server_negotiation.cc:408] Connection header received
1027 14:52:23.147875 (+ 1509us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:23.147881 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:23.147948 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:23.148037 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:23.150239 (+ 2202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.151250 (+ 1011us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.152618 (+ 1368us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.153124 (+ 506us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.157976 (+ 4852us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:53361: BlockingRecv error: recv got EOF from 127.0.104.3:53361 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:52:23.158936 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:23.145401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:53361)
1027 14:52:23.146190 (+ 789us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:23.146237 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:23.147610 (+ 1373us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:23.148322 (+ 712us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:23.148334 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:23.149095 (+ 761us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:23.150035 (+ 940us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.150062 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.151406 (+ 1344us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.151412 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:23.152488 (+ 1076us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.152500 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.152713 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.153362 (+ 649us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:23.153400 (+ 38us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:23.157733 (+ 4333us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":415,"threads_started":1}
W20251027 14:52:23.159696 10364 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:41623 (0 consecutive failures): Not authorized: Failed to ping master at 127.0.104.62:41623: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
I20251027 14:52:23.163990 10368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:23.149415 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:52330 (local address 127.0.104.62:41623)
1027 14:52:23.150074 (+ 659us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:23.150078 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:23.150093 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:52:23.150142 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:23.150148 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:23.150218 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:23.150350 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:23.152078 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.153325 (+ 1247us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.154939 (+ 1614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.155376 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.157634 (+ 2258us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:23.157688 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:23.157708 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:23.157745 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:23.161039 (+ 3294us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:23.161730 (+ 691us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:23.161738 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:23.161745 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:23.161858 (+ 113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:23.162167 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:23.162170 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:23.162172 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:23.162544 (+ 372us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:23.162796 (+ 252us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:23.163239 (+ 443us) server_negotiation.cc:300] Negotiation successful
1027 14:52:23.163606 (+ 367us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":568,"thread_start_us":471,"threads_started":1}
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:23.167797 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:23.160260 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:51541)
1027 14:52:23.160456 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:23.160476 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:23.160594 (+ 118us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:23.160887 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:23.160892 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:23.161258 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:23.161569 (+ 311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.161578 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.162910 (+ 1332us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.162914 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:23.163991 (+ 1077us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.164006 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.164167 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.164803 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:23.164826 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:23.167553 (+ 2727us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":64}
W20251027 14:52:23.167961 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:23.160422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:51541 (local address 127.0.104.62:41623)
1027 14:52:23.160576 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:23.160580 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:23.160595 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:52:23.160692 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:23.160696 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:23.160760 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:23.160834 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:23.161721 (+ 887us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.162774 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.164139 (+ 1365us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.164713 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.167766 (+ 3053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:51541: BlockingRecv error: recv got EOF from 127.0.104.3:51541 (error 108)
Metrics: {"server-negotiator.queue_time_us":40}
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:23.174882 10366 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:23.168461 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:37251)
1027 14:52:23.168622 (+ 161us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:23.168641 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:23.168738 (+ 97us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:23.169017 (+ 279us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:23.169022 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:23.169316 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:23.169614 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.169622 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.170717 (+ 1095us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.170721 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:23.171421 (+ 700us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.171431 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.171565 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.172166 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:23.172184 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:23.174674 (+ 2490us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":55}
W20251027 14:52:23.175068 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:23.168514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:37251 (local address 127.0.104.62:41623)
1027 14:52:23.168719 (+ 205us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:23.168724 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:23.168741 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:52:23.168856 (+ 115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:23.168860 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:23.168911 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:23.168983 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:23.169736 (+ 753us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.170595 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.171544 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.171954 (+ 410us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.174875 (+ 2921us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:37251: BlockingRecv error: recv got EOF from 127.0.104.3:37251 (error 108)
Metrics: {"server-negotiator.queue_time_us":115}
W20251027 14:52:23.175170 10364 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:23.304005 10369 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:23.296524 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:33531)
1027 14:52:23.297015 (+ 491us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:23.297030 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:23.297169 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:23.297493 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:23.297498 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:23.297990 (+ 492us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:23.298494 (+ 504us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.298508 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.299664 (+ 1156us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.299670 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:23.300450 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.300462 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.300629 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.301240 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:23.301258 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:23.303777 (+ 2519us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":253,"threads_started":1}
W20251027 14:52:23.304131 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:23.296648 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:33531 (local address 127.0.104.62:41623)
1027 14:52:23.296849 (+ 201us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:23.296856 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:23.297113 (+ 257us) server_negotiation.cc:408] Connection header received
1027 14:52:23.297286 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:23.297292 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:23.297360 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:23.297440 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:23.298641 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.299528 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.300583 (+ 1055us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.301004 (+ 421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.303967 (+ 2963us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:33531: BlockingRecv error: recv got EOF from 127.0.104.1:33531 (error 108)
Metrics: {"server-negotiator.queue_time_us":53}
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:23 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:23.752730 10370 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:23.744998 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:54703)
1027 14:52:23.745449 (+ 451us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:23.745463 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:23.745599 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:23.745923 (+ 324us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:23.745928 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:23.746357 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:23.746817 (+ 460us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.746827 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.748070 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.748077 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:23.749081 (+ 1004us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:23.749096 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.749270 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.749884 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:23.749903 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:23.752501 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":252,"threads_started":1}
W20251027 14:52:23.752912 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:23.745061 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:54703 (local address 127.0.104.62:41623)
1027 14:52:23.745239 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:23.745245 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:23.745549 (+ 304us) server_negotiation.cc:408] Connection header received
1027 14:52:23.745720 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:23.745726 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:23.745795 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:23.745877 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:23.746968 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.747916 (+ 948us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:23.749218 (+ 1302us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:23.749590 (+ 372us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:23.752700 (+ 3110us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:54703: BlockingRecv error: recv got EOF from 127.0.104.2:54703 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Oct 27 14:52:24 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:24 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:24.183557 10371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:24.175807 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:45551)
1027 14:52:24.176287 (+ 480us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:24.176302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:24.176493 (+ 191us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:24.176833 (+ 340us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:24.176838 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:24.177273 (+ 435us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:24.177685 (+ 412us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:24.177696 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.178827 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:24.178832 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:24.179616 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:24.179627 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.179792 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:24.180537 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:24.180556 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:24.183241 (+ 2685us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":265,"threads_started":1}
W20251027 14:52:24.183646 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:24.175945 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:45551 (local address 127.0.104.62:41623)
1027 14:52:24.176160 (+ 215us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:24.176167 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:24.176407 (+ 240us) server_negotiation.cc:408] Connection header received
1027 14:52:24.176620 (+ 213us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:24.176624 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:24.176685 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:24.176786 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:24.177835 (+ 1049us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.178681 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:24.179758 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.180179 (+ 421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:24.183455 (+ 3276us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:45551: BlockingRecv error: recv got EOF from 127.0.104.3:45551 (error 108)
Metrics: {"server-negotiator.queue_time_us":89}
Oct 27 14:52:24 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:24 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:24.312112 10372 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:24.304861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:56033)
1027 14:52:24.305307 (+ 446us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:24.305322 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:24.305455 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:24.305750 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:24.305756 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:24.306089 (+ 333us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:24.306441 (+ 352us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:24.306485 (+ 44us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.307708 (+ 1223us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:24.307713 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:24.308500 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:24.308512 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.308645 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:24.309324 (+ 679us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:24.309342 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:24.311837 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":241,"threads_started":1}
W20251027 14:52:24.312300 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:24.304904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:56033 (local address 127.0.104.62:41623)
1027 14:52:24.305074 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:24.305081 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:24.305396 (+ 315us) server_negotiation.cc:408] Connection header received
1027 14:52:24.305563 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:24.305569 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:24.305630 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:24.305717 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:24.306617 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.307583 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:24.308635 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.309084 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:24.312059 (+ 2975us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:56033: BlockingRecv error: recv got EOF from 127.0.104.1:56033 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Oct 27 14:52:24 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:24 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:24.761081 10373 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:24.753612 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:51045)
1027 14:52:24.754132 (+ 520us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:24.754147 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:24.754260 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:24.754578 (+ 318us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:24.754581 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:24.754893 (+ 312us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:24.755229 (+ 336us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:24.755244 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.756333 (+ 1089us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:24.756337 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:24.757080 (+ 743us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:24.757091 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.757205 (+ 114us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:24.757813 (+ 608us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:24.757831 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:24.760798 (+ 2967us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":426,"thread_start_us":312,"threads_started":1}
W20251027 14:52:24.761183 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:24.753721 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:51045 (local address 127.0.104.62:41623)
1027 14:52:24.753889 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:24.753894 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:24.754222 (+ 328us) server_negotiation.cc:408] Connection header received
1027 14:52:24.754394 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:24.754399 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:24.754465 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:24.754550 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:24.755344 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.756221 (+ 877us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:24.757223 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:24.757602 (+ 379us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:24.760987 (+ 3385us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:51045: BlockingRecv error: recv got EOF from 127.0.104.2:51045 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
Oct 27 14:52:25 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:25 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:25.191835 10374 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:25.184427 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:45739)
1027 14:52:25.184926 (+ 499us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:25.184942 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:25.185073 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:25.185426 (+ 353us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:25.185432 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:25.185768 (+ 336us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:25.186100 (+ 332us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:25.186117 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.187321 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:25.187325 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:25.188054 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:25.188065 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.188306 (+ 241us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:25.189008 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:25.189027 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:25.191625 (+ 2598us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":398,"thread_start_us":272,"threads_started":1}
W20251027 14:52:25.192011 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:25.184643 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:45739 (local address 127.0.104.62:41623)
1027 14:52:25.184814 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:25.184820 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:25.185020 (+ 200us) server_negotiation.cc:408] Connection header received
1027 14:52:25.185187 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:25.185194 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:25.185289 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:25.185375 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:25.186289 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.187171 (+ 882us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:25.188268 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.188794 (+ 526us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:25.191820 (+ 3026us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:45739: BlockingRecv error: recv got EOF from 127.0.104.3:45739 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Oct 27 14:52:25 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:25 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:25.320708 10375 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:25.313138 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:55551)
1027 14:52:25.313624 (+ 486us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:25.313642 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:25.313784 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:25.314077 (+ 293us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:25.314080 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:25.314474 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:25.314898 (+ 424us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:25.314909 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.316198 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:25.316204 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:25.317080 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:25.317096 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.317234 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:25.317894 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:25.317912 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:25.320484 (+ 2572us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":384,"thread_start_us":283,"threads_started":1}
W20251027 14:52:25.320885 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:25.313255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:55551 (local address 127.0.104.62:41623)
1027 14:52:25.313436 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:25.313441 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:25.313720 (+ 279us) server_negotiation.cc:408] Connection header received
1027 14:52:25.313888 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:25.313892 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:25.313957 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:25.314048 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:25.315052 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.316058 (+ 1006us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:25.317229 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.317644 (+ 415us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:25.320686 (+ 3042us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:55551: BlockingRecv error: recv got EOF from 127.0.104.1:55551 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
Oct 27 14:52:25 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:25 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:25.769554 10376 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:25.762034 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:58999)
1027 14:52:25.762524 (+ 490us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:25.762540 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:25.762662 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:25.762937 (+ 275us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:25.762941 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:25.763302 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:25.763730 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:25.763740 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.764842 (+ 1102us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:25.764847 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:25.765601 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:25.765613 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.765724 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:25.766440 (+ 716us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:25.766459 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:25.769326 (+ 2867us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":273,"threads_started":1}
W20251027 14:52:25.769731 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:25.762124 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:58999 (local address 127.0.104.62:41623)
1027 14:52:25.762268 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:25.762273 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:25.762609 (+ 336us) server_negotiation.cc:408] Connection header received
1027 14:52:25.762773 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:25.762777 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:25.762838 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:25.762908 (+ 70us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:25.763840 (+ 932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.764723 (+ 883us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:25.765725 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:25.766105 (+ 380us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:25.769498 (+ 3393us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:58999: BlockingRecv error: recv got EOF from 127.0.104.2:58999 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 27 14:52:26 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:26 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:26.200202 10377 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:26.192756 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:47861)
1027 14:52:26.193189 (+ 433us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:26.193203 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:26.193343 (+ 140us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:26.193655 (+ 312us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:26.193660 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:26.194053 (+ 393us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:26.194470 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:26.194480 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.195704 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:26.195708 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:26.196523 (+ 815us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:26.196536 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.196671 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:26.197297 (+ 626us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:26.197316 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:26.199898 (+ 2582us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":239,"threads_started":1}
W20251027 14:52:26.200341 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:26.192834 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:47861 (local address 127.0.104.62:41623)
1027 14:52:26.193003 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:26.193009 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:26.193281 (+ 272us) server_negotiation.cc:408] Connection header received
1027 14:52:26.193464 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:26.193469 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:26.193529 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:26.193608 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:26.194612 (+ 1004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.195558 (+ 946us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:26.196655 (+ 1097us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.197057 (+ 402us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:26.200121 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:47861: BlockingRecv error: recv got EOF from 127.0.104.3:47861 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 27 14:52:26 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:26 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:26.329144 10378 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:26.321709 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:54001)
1027 14:52:26.322163 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:26.322179 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:26.322362 (+ 183us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:26.322699 (+ 337us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:26.322704 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:26.323141 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:26.323605 (+ 464us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:26.323621 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.324806 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:26.324810 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:26.325557 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:26.325569 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.325682 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:26.326373 (+ 691us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:26.326391 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:26.328933 (+ 2542us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":246,"threads_started":1}
W20251027 14:52:26.329370 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:26.321806 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:54001 (local address 127.0.104.62:41623)
1027 14:52:26.321990 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:26.321996 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:26.322321 (+ 325us) server_negotiation.cc:408] Connection header received
1027 14:52:26.322465 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:26.322471 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:26.322559 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:26.322649 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:26.323758 (+ 1109us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.324689 (+ 931us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:26.325669 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.326054 (+ 385us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:26.329149 (+ 3095us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:54001: BlockingRecv error: recv got EOF from 127.0.104.1:54001 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
Oct 27 14:52:26 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:26 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:26.778694 10379 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:26.770566 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:40729)
1027 14:52:26.771070 (+ 504us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:26.771089 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:26.771227 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:26.771673 (+ 446us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:26.771678 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:26.772162 (+ 484us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:26.772743 (+ 581us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:26.772757 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.773988 (+ 1231us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:26.773992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:26.774723 (+ 731us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:26.774734 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.774863 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:26.775575 (+ 712us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:26.775593 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:26.778488 (+ 2895us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":276,"threads_started":1}
W20251027 14:52:26.778959 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:26.770665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:40729 (local address 127.0.104.62:41623)
1027 14:52:26.770844 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:26.770851 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:26.771183 (+ 332us) server_negotiation.cc:408] Connection header received
1027 14:52:26.771439 (+ 256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:26.771445 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:26.771529 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:26.771628 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:26.772889 (+ 1261us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.773855 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:26.774856 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:26.775312 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:26.778720 (+ 3408us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:40729: BlockingRecv error: recv got EOF from 127.0.104.2:40729 (error 108)
Metrics: {"server-negotiator.queue_time_us":52}
Oct 27 14:52:27 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:27 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:27.208549 10380 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:27.201149 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:33471)
1027 14:52:27.201680 (+ 531us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:27.201696 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:27.201813 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:27.202134 (+ 321us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:27.202138 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:27.202538 (+ 400us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:27.202907 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:27.202917 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.204095 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:27.204100 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:27.204899 (+ 799us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:27.204911 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.205028 (+ 117us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:27.205638 (+ 610us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:27.205656 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:27.208287 (+ 2631us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":270,"threads_started":1}
W20251027 14:52:27.208727 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:27.201179 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:33471 (local address 127.0.104.62:41623)
1027 14:52:27.201387 (+ 208us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:27.201392 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:27.201784 (+ 392us) server_negotiation.cc:408] Connection header received
1027 14:52:27.201906 (+ 122us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:27.201911 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:27.201977 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:27.202060 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:27.203051 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.203956 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:27.205062 (+ 1106us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.205397 (+ 335us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:27.208516 (+ 3119us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:33471: BlockingRecv error: recv got EOF from 127.0.104.3:33471 (error 108)
Metrics: {"server-negotiator.queue_time_us":95}
Oct 27 14:52:27 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:27 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:27.337590 10381 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:27.330009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:38381)
1027 14:52:27.330473 (+ 464us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:27.330489 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:27.330613 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:27.330916 (+ 303us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:27.330922 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:27.331312 (+ 390us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:27.331684 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:27.331694 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.332920 (+ 1226us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:27.332925 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:27.333801 (+ 876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:27.333818 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.333951 (+ 133us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:27.334613 (+ 662us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:27.334634 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:27.337368 (+ 2734us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":241,"threads_started":1}
W20251027 14:52:27.337762 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:27.330100 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:38381 (local address 127.0.104.62:41623)
1027 14:52:27.330300 (+ 200us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:27.330306 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:27.330556 (+ 250us) server_negotiation.cc:408] Connection header received
1027 14:52:27.330728 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:27.330733 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:27.330796 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:27.330875 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:27.331818 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.332755 (+ 937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:27.333953 (+ 1198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.334415 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:27.337559 (+ 3144us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:38381: BlockingRecv error: recv got EOF from 127.0.104.1:38381 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 27 14:52:27 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:27 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:27.791607 10382 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:27.779838 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:55601)
1027 14:52:27.780303 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:27.780319 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:27.780483 (+ 164us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:27.780774 (+ 291us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:27.780778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:27.781157 (+ 379us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:27.781541 (+ 384us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:27.781551 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.782746 (+ 1195us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:27.782750 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:27.783550 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:27.783564 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.783704 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:27.788789 (+ 5085us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:27.788810 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:27.791305 (+ 2495us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":235,"threads_started":1}
W20251027 14:52:27.791764 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:27.780067 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:55601 (local address 127.0.104.62:41623)
1027 14:52:27.780295 (+ 228us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:27.780300 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:27.780398 (+ 98us) server_negotiation.cc:408] Connection header received
1027 14:52:27.780583 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:27.780587 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:27.780661 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:27.780743 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:27.781692 (+ 949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.782614 (+ 922us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:27.783706 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:27.784138 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:27.791549 (+ 7411us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:55601: BlockingRecv error: recv got EOF from 127.0.104.2:55601 (error 108)
Metrics: {"server-negotiator.queue_time_us":111}
Oct 27 14:52:28 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:28 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:28.216873 10383 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:28.209576 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:43807)
1027 14:52:28.210278 (+ 702us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:28.210294 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:28.210430 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:28.210696 (+ 266us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:28.210700 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:28.211068 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:28.211443 (+ 375us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:28.211454 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.212513 (+ 1059us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:28.212518 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:28.213264 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:28.213274 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.213390 (+ 116us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:28.214115 (+ 725us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:28.214135 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:28.216659 (+ 2524us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":601,"thread_start_us":485,"threads_started":1}
W20251027 14:52:28.217128 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:28.209676 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:43807 (local address 127.0.104.62:41623)
1027 14:52:28.209978 (+ 302us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:28.209984 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:28.210379 (+ 395us) server_negotiation.cc:408] Connection header received
1027 14:52:28.210521 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:28.210525 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:28.210593 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:28.210671 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:28.211553 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.212412 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:28.213372 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.213797 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:28.216903 (+ 3106us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:43807: BlockingRecv error: recv got EOF from 127.0.104.3:43807 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 27 14:52:28 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:28 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:28.346828 10384 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:28.338508 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:34137)
1027 14:52:28.338966 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:28.338985 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:28.339150 (+ 165us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:28.339500 (+ 350us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:28.339505 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:28.339942 (+ 437us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:28.340496 (+ 554us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:28.340511 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.342131 (+ 1620us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:28.342138 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:28.343195 (+ 1057us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:28.343211 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.343366 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:28.344070 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:28.344088 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:28.346633 (+ 2545us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":336,"thread_start_us":228,"threads_started":1}
W20251027 14:52:28.347064 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:28.338564 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:34137 (local address 127.0.104.62:41623)
1027 14:52:28.338742 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:28.338748 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:28.339049 (+ 301us) server_negotiation.cc:408] Connection header received
1027 14:52:28.339278 (+ 229us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:28.339284 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:28.339359 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:28.339463 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:28.340667 (+ 1204us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.341965 (+ 1298us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:28.343329 (+ 1364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.343792 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:28.346838 (+ 3046us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:34137: BlockingRecv error: recv got EOF from 127.0.104.1:34137 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
Oct 27 14:52:28 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:28 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:28.800087 10385 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:28.792645 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:56575)
1027 14:52:28.793108 (+ 463us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:28.793125 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:28.793258 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:28.793571 (+ 313us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:28.793577 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:28.793945 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:28.794352 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:28.794362 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.795470 (+ 1108us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:28.795476 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:28.796252 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:28.796264 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.796444 (+ 180us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:28.797114 (+ 670us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:28.797135 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:28.799858 (+ 2723us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":343,"thread_start_us":239,"threads_started":1}
W20251027 14:52:28.800347 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:28.792685 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:56575 (local address 127.0.104.62:41623)
1027 14:52:28.792836 (+ 151us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:28.792843 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:28.793200 (+ 357us) server_negotiation.cc:408] Connection header received
1027 14:52:28.793384 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:28.793390 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:28.793453 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:28.793528 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:28.794491 (+ 963us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.795339 (+ 848us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:28.796418 (+ 1079us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:28.796857 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:28.800123 (+ 3266us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:56575: BlockingRecv error: recv got EOF from 127.0.104.2:56575 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
Oct 27 14:52:29 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:29 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:29.225178 10386 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:29.217814 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:54223)
1027 14:52:29.218406 (+ 592us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:29.218422 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:29.218563 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:29.218894 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:29.218899 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:29.219261 (+ 362us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:29.219656 (+ 395us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:29.219666 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.220859 (+ 1193us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:29.220864 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:29.221589 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:29.221600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.221749 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:29.222412 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:29.222430 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:29.224967 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":486,"thread_start_us":365,"threads_started":1}
W20251027 14:52:29.225373 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:29.217888 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:54223 (local address 127.0.104.62:41623)
1027 14:52:29.218106 (+ 218us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:29.218112 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:29.218516 (+ 404us) server_negotiation.cc:408] Connection header received
1027 14:52:29.218685 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:29.218690 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:29.218753 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:29.218834 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:29.219800 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.220728 (+ 928us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:29.221725 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.222152 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:29.225172 (+ 3020us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:54223: BlockingRecv error: recv got EOF from 127.0.104.3:54223 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Oct 27 14:52:29 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:29 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:29.355073 10387 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:29.347655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:57803)
1027 14:52:29.348144 (+ 489us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:29.348160 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:29.348290 (+ 130us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:29.348572 (+ 282us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:29.348576 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:29.348957 (+ 381us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:29.349346 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:29.349356 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.350763 (+ 1407us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:29.350768 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:29.351677 (+ 909us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:29.351690 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.351803 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:29.352483 (+ 680us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:29.352501 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:29.354849 (+ 2348us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":391,"thread_start_us":209,"threads_started":1}
W20251027 14:52:29.355216 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:29.347752 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:57803 (local address 127.0.104.62:41623)
1027 14:52:29.347958 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:29.347964 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:29.348234 (+ 270us) server_negotiation.cc:408] Connection header received
1027 14:52:29.348402 (+ 168us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:29.348406 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:29.348469 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:29.348546 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:29.349458 (+ 912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.350614 (+ 1156us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:29.351809 (+ 1195us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.352169 (+ 360us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:29.355020 (+ 2851us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:57803: BlockingRecv error: recv got EOF from 127.0.104.1:57803 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Oct 27 14:52:29 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:29 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:29.808367 10388 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:29.800953 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:48945)
1027 14:52:29.801530 (+ 577us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:29.801544 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:29.801690 (+ 146us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:29.802001 (+ 311us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:29.802007 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:29.802437 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:29.802802 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:29.802811 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.803963 (+ 1152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:29.803969 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:29.804769 (+ 800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:29.804781 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.804915 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:29.805516 (+ 601us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:29.805535 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:29.808150 (+ 2615us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":450,"thread_start_us":250,"threads_started":1}
W20251027 14:52:29.808605 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:29.801007 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:48945 (local address 127.0.104.62:41623)
1027 14:52:29.801237 (+ 230us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:29.801244 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:29.801625 (+ 381us) server_negotiation.cc:408] Connection header received
1027 14:52:29.801803 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:29.801809 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:29.801872 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:29.801949 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:29.802936 (+ 987us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.803825 (+ 889us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:29.804893 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:29.805302 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:29.808332 (+ 3030us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:48945: BlockingRecv error: recv got EOF from 127.0.104.2:48945 (error 108)
Metrics: {"server-negotiator.queue_time_us":112}
Oct 27 14:52:30 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:30 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:30.233377 10389 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:30.226093 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:36595)
1027 14:52:30.226651 (+ 558us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:30.226667 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:30.226780 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:30.227064 (+ 284us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:30.227068 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:30.227446 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:30.227818 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:30.227827 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.228998 (+ 1171us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:30.229002 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:30.229717 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:30.229727 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.229840 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:30.230476 (+ 636us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:30.230494 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:30.233167 (+ 2673us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":455,"thread_start_us":248,"threads_started":1}
W20251027 14:52:30.233599 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:30.226235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:36595 (local address 127.0.104.62:41623)
1027 14:52:30.226387 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:30.226394 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:30.226742 (+ 348us) server_negotiation.cc:408] Connection header received
1027 14:52:30.226868 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:30.226872 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:30.226935 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:30.227026 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:30.227968 (+ 942us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.228871 (+ 903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:30.229825 (+ 954us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.230118 (+ 293us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:30.233392 (+ 3274us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:36595: BlockingRecv error: recv got EOF from 127.0.104.3:36595 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Oct 27 14:52:30 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:30 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:30.363608 10390 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:30.356140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:37739)
1027 14:52:30.356606 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:30.356625 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:30.356780 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:30.357097 (+ 317us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:30.357102 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:30.357526 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:30.357932 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:30.357952 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.359278 (+ 1326us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:30.359283 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:30.360008 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:30.360019 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.360155 (+ 136us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:30.360808 (+ 653us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:30.360826 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:30.363403 (+ 2577us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":248,"threads_started":1}
W20251027 14:52:30.363843 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:30.356428 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:37739 (local address 127.0.104.62:41623)
1027 14:52:30.356606 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:30.356612 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:30.356706 (+ 94us) server_negotiation.cc:408] Connection header received
1027 14:52:30.356899 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:30.356904 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:30.356973 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:30.357053 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:30.358088 (+ 1035us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.359150 (+ 1062us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:30.360140 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.360587 (+ 447us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:30.363618 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:37739: BlockingRecv error: recv got EOF from 127.0.104.1:37739 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
Oct 27 14:52:30 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:30 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:30.816692 10391 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:30.809295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:56359)
1027 14:52:30.809714 (+ 419us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:30.809730 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:30.809844 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:30.810186 (+ 342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:30.810190 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:30.810538 (+ 348us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:30.810908 (+ 370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:30.810925 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.812263 (+ 1338us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:30.812267 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:30.813030 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:30.813041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.813160 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:30.813793 (+ 633us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:30.813810 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:30.816439 (+ 2629us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":334,"thread_start_us":232,"threads_started":1}
W20251027 14:52:30.816905 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:30.809362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:56359 (local address 127.0.104.62:41623)
1027 14:52:30.809510 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:30.809517 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:30.809798 (+ 281us) server_negotiation.cc:408] Connection header received
1027 14:52:30.809974 (+ 176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:30.809979 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:30.810045 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:30.810146 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:30.811064 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.812154 (+ 1090us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:30.813177 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:30.813562 (+ 385us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:30.816676 (+ 3114us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:56359: BlockingRecv error: recv got EOF from 127.0.104.2:56359 (error 108)
Metrics: {"server-negotiator.queue_time_us":33}
Oct 27 14:52:31 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:31 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:31.242179 10392 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:31.234355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:56935)
1027 14:52:31.234796 (+ 441us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:31.234811 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:31.234979 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:31.235345 (+ 366us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:31.235350 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:31.235718 (+ 368us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:31.236091 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:31.236109 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.237343 (+ 1234us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:31.237347 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:31.238065 (+ 718us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:31.238076 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.238214 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:31.239346 (+ 1132us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:31.239364 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:31.241967 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":241,"threads_started":1}
W20251027 14:52:31.242422 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:31.234429 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:56935 (local address 127.0.104.62:41623)
1027 14:52:31.234597 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:31.234604 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:31.234899 (+ 295us) server_negotiation.cc:408] Connection header received
1027 14:52:31.235103 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:31.235108 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:31.235188 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:31.235292 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:31.236250 (+ 958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.237209 (+ 959us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:31.238203 (+ 994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.238684 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:31.242195 (+ 3511us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:56935: BlockingRecv error: recv got EOF from 127.0.104.3:56935 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 27 14:52:31 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:31 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:31.371819 10393 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:31.364498 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:35239)
1027 14:52:31.364900 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:31.364917 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:31.365031 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:31.365325 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:31.365329 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:31.365699 (+ 370us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:31.366088 (+ 389us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:31.366097 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.367222 (+ 1125us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:31.367226 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:31.367971 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:31.367982 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.368122 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:31.368806 (+ 684us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:31.368824 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:31.371592 (+ 2768us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":205,"threads_started":1}
W20251027 14:52:31.372071 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:31.364553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:35239 (local address 127.0.104.62:41623)
1027 14:52:31.364748 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:31.364754 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:31.364988 (+ 234us) server_negotiation.cc:408] Connection header received
1027 14:52:31.365136 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:31.365140 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:31.365215 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:31.365295 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:31.366221 (+ 926us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.367087 (+ 866us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:31.368084 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.368488 (+ 404us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:31.371822 (+ 3334us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:35239: BlockingRecv error: recv got EOF from 127.0.104.1:35239 (error 108)
Metrics: {"server-negotiator.queue_time_us":58}
Oct 27 14:52:31 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:31 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:31.825323 10394 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:31.817588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:38127)
1027 14:52:31.818162 (+ 574us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:31.818183 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:31.818314 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:31.818694 (+ 380us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:31.818699 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:31.819160 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:31.819595 (+ 435us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:31.819610 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.821038 (+ 1428us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:31.821042 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:31.821787 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:31.821797 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.821935 (+ 138us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:31.822534 (+ 599us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:31.822552 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:31.825118 (+ 2566us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":293,"threads_started":1}
W20251027 14:52:31.825534 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:31.817645 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:38127 (local address 127.0.104.62:41623)
1027 14:52:31.817793 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:31.817797 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:31.818258 (+ 461us) server_negotiation.cc:408] Connection header received
1027 14:52:31.818452 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:31.818458 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:31.818523 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:31.818615 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:31.819755 (+ 1140us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.820899 (+ 1144us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:31.821909 (+ 1010us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:31.822266 (+ 357us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:31.825330 (+ 3064us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:38127: BlockingRecv error: recv got EOF from 127.0.104.2:38127 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 27 14:52:32 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:32 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:32.250327 10395 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:32.243152 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:59429)
1027 14:52:32.243636 (+ 484us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:32.243651 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:32.243763 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:32.244104 (+ 341us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:32.244108 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:32.244609 (+ 501us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:32.244991 (+ 382us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:32.245001 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.246132 (+ 1131us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:32.246137 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:32.246884 (+ 747us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:32.246894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.247033 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:32.247614 (+ 581us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:32.247633 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:32.250118 (+ 2485us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":392,"thread_start_us":239,"threads_started":1}
W20251027 14:52:32.250578 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:32.243225 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:59429 (local address 127.0.104.62:41623)
1027 14:52:32.243407 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:32.243413 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:32.243729 (+ 316us) server_negotiation.cc:408] Connection header received
1027 14:52:32.243891 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:32.243896 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:32.243958 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:32.244060 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:32.245122 (+ 1062us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.246001 (+ 879us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:32.246989 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.247345 (+ 356us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:32.250356 (+ 3011us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:59429: BlockingRecv error: recv got EOF from 127.0.104.3:59429 (error 108)
Metrics: {"server-negotiator.queue_time_us":51}
Oct 27 14:52:32 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:32 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:32.380122 10396 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:32.372850 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:55555)
1027 14:52:32.373298 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:32.373314 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:32.373431 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:32.373705 (+ 274us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:32.373708 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:32.374074 (+ 366us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:32.374421 (+ 347us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:32.374430 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.375650 (+ 1220us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:32.375655 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:32.376599 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:32.376612 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.376758 (+ 146us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:32.377371 (+ 613us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:32.377389 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:32.379926 (+ 2537us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":237,"threads_started":1}
W20251027 14:52:32.380302 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:32.372909 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:55555 (local address 127.0.104.62:41623)
1027 14:52:32.373094 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:32.373101 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:32.373394 (+ 293us) server_negotiation.cc:408] Connection header received
1027 14:52:32.373522 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:32.373526 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:32.373599 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:32.373676 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:32.374538 (+ 862us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.375525 (+ 987us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:32.376738 (+ 1213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.377111 (+ 373us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:32.380097 (+ 2986us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:55555: BlockingRecv error: recv got EOF from 127.0.104.1:55555 (error 108)
Metrics: {"server-negotiator.queue_time_us":56}
Oct 27 14:52:32 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:32 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:32.837711 10397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:32.826393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:37071)
1027 14:52:32.826953 (+ 560us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:32.826968 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:32.827111 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:32.827458 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:32.827464 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:32.827842 (+ 378us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:32.828192 (+ 350us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:32.828201 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.829361 (+ 1160us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:32.829365 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:32.830141 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:32.830154 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.830293 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:32.834797 (+ 4504us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:32.834818 (+ 21us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:32.837506 (+ 2688us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":464,"thread_start_us":339,"threads_started":1}
W20251027 14:52:32.837999 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:32.826533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:37071 (local address 127.0.104.62:41623)
1027 14:52:32.826722 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:32.826728 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:32.827056 (+ 328us) server_negotiation.cc:408] Connection header received
1027 14:52:32.827233 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:32.827238 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:32.827328 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:32.827409 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:32.828332 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.829233 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:32.830273 (+ 1040us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:32.830707 (+ 434us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:32.837760 (+ 7053us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:37071: BlockingRecv error: recv got EOF from 127.0.104.2:37071 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
Oct 27 14:52:33 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:33 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:33.258424 10398 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:33.251292 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:53915)
1027 14:52:33.251784 (+ 492us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:33.251800 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:33.251910 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:33.252167 (+ 257us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:33.252171 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:33.252551 (+ 380us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:33.252927 (+ 376us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:33.252937 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.254031 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:33.254035 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:33.254809 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:33.254819 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.254939 (+ 120us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:33.255550 (+ 611us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:33.255568 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:33.258220 (+ 2652us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":242,"threads_started":1}
W20251027 14:52:33.258720 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:33.251407 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:53915 (local address 127.0.104.62:41623)
1027 14:52:33.251586 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:33.251590 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:33.251861 (+ 271us) server_negotiation.cc:408] Connection header received
1027 14:52:33.251995 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:33.251999 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:33.252061 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:33.252139 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:33.253055 (+ 916us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.253901 (+ 846us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:33.254959 (+ 1058us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.255313 (+ 354us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:33.258470 (+ 3157us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:53915: BlockingRecv error: recv got EOF from 127.0.104.3:53915 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
Oct 27 14:52:33 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:33 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:33.388557 10399 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:33.381043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:59879)
1027 14:52:33.381499 (+ 456us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:33.381514 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:33.381696 (+ 182us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:33.382027 (+ 331us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:33.382031 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:33.382498 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:33.382959 (+ 461us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:33.382972 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.384251 (+ 1279us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:33.384255 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:33.384993 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:33.385004 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.385115 (+ 111us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:33.385729 (+ 614us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:33.385747 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:33.388287 (+ 2540us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":257,"threads_started":1}
W20251027 14:52:33.388693 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:33.381158 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:59879 (local address 127.0.104.62:41623)
1027 14:52:33.381369 (+ 211us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:33.381377 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:33.381596 (+ 219us) server_negotiation.cc:408] Connection header received
1027 14:52:33.381820 (+ 224us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:33.381825 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:33.381888 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:33.381992 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:33.383091 (+ 1099us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.384120 (+ 1029us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:33.385102 (+ 982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.385438 (+ 336us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:33.388528 (+ 3090us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:59879: BlockingRecv error: recv got EOF from 127.0.104.1:59879 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Oct 27 14:52:33 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:33 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:33.846139 10400 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:33.838675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:36371)
1027 14:52:33.839155 (+ 480us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:33.839170 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:33.839287 (+ 117us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:33.839581 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:33.839584 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:33.839985 (+ 401us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:33.840402 (+ 417us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:33.840416 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.841641 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:33.841646 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:33.842384 (+ 738us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:33.842396 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.842533 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:33.843327 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:33.843346 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:33.845904 (+ 2558us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":387,"thread_start_us":282,"threads_started":1}
W20251027 14:52:33.846349 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:33.838732 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:36371 (local address 127.0.104.62:41623)
1027 14:52:33.838884 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:33.838889 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:33.839273 (+ 384us) server_negotiation.cc:408] Connection header received
1027 14:52:33.839386 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:33.839390 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:33.839461 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:33.839548 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:33.840538 (+ 990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.841466 (+ 928us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:33.842518 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:33.843000 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:33.846123 (+ 3123us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:36371: BlockingRecv error: recv got EOF from 127.0.104.2:36371 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
Oct 27 14:52:34 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:34 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:34.266542 10401 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:34.259355 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:38685)
1027 14:52:34.259810 (+ 455us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:34.259825 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:34.259944 (+ 119us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:34.260205 (+ 261us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:34.260208 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:34.260597 (+ 389us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:34.261003 (+ 406us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:34.261012 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.262159 (+ 1147us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:34.262162 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:34.262873 (+ 711us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:34.262882 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.262992 (+ 110us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:34.263651 (+ 659us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:34.263669 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:34.266318 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":358,"thread_start_us":240,"threads_started":1}
W20251027 14:52:34.266680 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:34.259432 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:38685 (local address 127.0.104.62:41623)
1027 14:52:34.259608 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:34.259613 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:34.259891 (+ 278us) server_negotiation.cc:408] Connection header received
1027 14:52:34.260035 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:34.260039 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:34.260098 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:34.260178 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:34.261156 (+ 978us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.262052 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:34.262976 (+ 924us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.263362 (+ 386us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:34.266491 (+ 3129us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:38685: BlockingRecv error: recv got EOF from 127.0.104.3:38685 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
Oct 27 14:52:34 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:34 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:34.396880 10402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:34.389426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:44781)
1027 14:52:34.389897 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:34.389913 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:34.390037 (+ 124us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:34.390331 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:34.390335 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:34.390679 (+ 344us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:34.391072 (+ 393us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:34.391082 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.392331 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:34.392335 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:34.393144 (+ 809us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:34.393158 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.393281 (+ 123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:34.393902 (+ 621us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:34.393920 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:34.396547 (+ 2627us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":378,"thread_start_us":238,"threads_started":1}
W20251027 14:52:34.397049 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:34.389533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:44781 (local address 127.0.104.62:41623)
1027 14:52:34.389775 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:34.389781 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:34.390017 (+ 236us) server_negotiation.cc:408] Connection header received
1027 14:52:34.390134 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:34.390139 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:34.390203 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:34.390297 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:34.391240 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.392182 (+ 942us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:34.393289 (+ 1107us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.393639 (+ 350us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:34.396775 (+ 3136us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:44781: BlockingRecv error: recv got EOF from 127.0.104.1:44781 (error 108)
Metrics: {"server-negotiator.queue_time_us":123}
Oct 27 14:52:34 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:34 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:34.854701 10403 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:34.847020 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:38475)
1027 14:52:34.847479 (+ 459us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:34.847526 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:34.847661 (+ 135us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:34.847981 (+ 320us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:34.847987 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:34.848341 (+ 354us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:34.848856 (+ 515us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:34.848869 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.850007 (+ 1138us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:34.850013 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:34.850800 (+ 787us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:34.850812 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.850949 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:34.851588 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:34.851606 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:34.854367 (+ 2761us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":241,"threads_started":1}
W20251027 14:52:34.854806 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:34.847137 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:38475 (local address 127.0.104.62:41623)
1027 14:52:34.847336 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:34.847343 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:34.847606 (+ 263us) server_negotiation.cc:408] Connection header received
1027 14:52:34.847787 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:34.847793 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:34.847857 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:34.847938 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:34.849002 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.849865 (+ 863us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:34.850940 (+ 1075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:34.851381 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:34.854602 (+ 3221us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:38475: BlockingRecv error: recv got EOF from 127.0.104.2:38475 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Oct 27 14:52:35 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:35 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:35.274422 10404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:35.267432 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:54999)
1027 14:52:35.267884 (+ 452us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:35.267900 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:35.268008 (+ 108us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:35.268259 (+ 251us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:35.268263 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:35.268650 (+ 387us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:35.269015 (+ 365us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:35.269024 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.270118 (+ 1094us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:35.270122 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:35.270859 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:35.270869 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.270984 (+ 115us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:35.271623 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:35.271641 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:35.274226 (+ 2585us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":233,"threads_started":1}
W20251027 14:52:35.274672 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:35.267479 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:54999 (local address 127.0.104.62:41623)
1027 14:52:35.267669 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:35.267675 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:35.267965 (+ 290us) server_negotiation.cc:408] Connection header received
1027 14:52:35.268091 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:35.268095 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:35.268155 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:35.268233 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:35.269129 (+ 896us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.269999 (+ 870us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:35.270960 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.271308 (+ 348us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:35.274447 (+ 3139us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:54999: BlockingRecv error: recv got EOF from 127.0.104.3:54999 (error 108)
Metrics: {"server-negotiator.queue_time_us":80}
Oct 27 14:52:35 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:35 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:35.405085 10405 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:35.397760 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:45449)
1027 14:52:35.398372 (+ 612us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:35.398387 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:35.398498 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:35.398768 (+ 270us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:35.398772 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:35.399133 (+ 361us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:35.399476 (+ 343us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:35.399484 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.400588 (+ 1104us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:35.400592 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:35.401378 (+ 786us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:35.401389 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.401555 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:35.402152 (+ 597us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:35.402170 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:35.404844 (+ 2674us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":500,"thread_start_us":343,"threads_started":1}
W20251027 14:52:35.405208 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:35.397817 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:45449 (local address 127.0.104.62:41623)
1027 14:52:35.398023 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:35.398030 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:35.398455 (+ 425us) server_negotiation.cc:408] Connection header received
1027 14:52:35.398603 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:35.398607 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:35.398666 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:35.398739 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:35.399605 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.400450 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:35.401506 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.401964 (+ 458us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:35.405030 (+ 3066us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:45449: BlockingRecv error: recv got EOF from 127.0.104.1:45449 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
Oct 27 14:52:35 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:35 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:35.863286 10406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:35.855685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:48739)
1027 14:52:35.856174 (+ 489us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:35.856194 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:35.856356 (+ 162us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:35.856814 (+ 458us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:35.856818 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:35.857280 (+ 462us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:35.857758 (+ 478us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:35.857774 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.858976 (+ 1202us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:35.858980 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:35.859709 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:35.859730 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.859889 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:35.860532 (+ 643us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:35.860551 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:35.863076 (+ 2525us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":404,"thread_start_us":278,"threads_started":1}
W20251027 14:52:35.863499 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:35.855811 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:48739 (local address 127.0.104.62:41623)
1027 14:52:35.855983 (+ 172us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:35.855989 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:35.856297 (+ 308us) server_negotiation.cc:408] Connection header received
1027 14:52:35.856561 (+ 264us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:35.856568 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:35.856653 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:35.856764 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:35.857932 (+ 1168us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.858851 (+ 919us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:35.859862 (+ 1011us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:35.860232 (+ 370us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:35.863256 (+ 3024us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:48739: BlockingRecv error: recv got EOF from 127.0.104.2:48739 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
Oct 27 14:52:36 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:36 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:36.282450 10407 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:36.275197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:41171)
1027 14:52:36.275667 (+ 470us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:36.275683 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:36.275806 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:36.276077 (+ 271us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:36.276082 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:36.276511 (+ 429us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:36.276884 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:36.276894 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.278028 (+ 1134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:36.278032 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:36.278786 (+ 754us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:36.278796 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.278930 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:36.279560 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:36.279578 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:36.282181 (+ 2603us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":364,"thread_start_us":249,"threads_started":1}
W20251027 14:52:36.282759 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:36.275279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:41171 (local address 127.0.104.62:41623)
1027 14:52:36.275490 (+ 211us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:36.275496 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:36.275756 (+ 260us) server_negotiation.cc:408] Connection header received
1027 14:52:36.275898 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:36.275902 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:36.275963 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:36.276039 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:36.277016 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.277908 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:36.278888 (+ 980us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.279240 (+ 352us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:36.282405 (+ 3165us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:41171: BlockingRecv error: recv got EOF from 127.0.104.3:41171 (error 108)
Metrics: {"server-negotiator.queue_time_us":85}
Oct 27 14:52:36 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:36 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:36.413419 10408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:36.406125 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:39133)
1027 14:52:36.406563 (+ 438us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:36.406582 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:36.406723 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:36.407039 (+ 316us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:36.407044 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:36.407427 (+ 383us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:36.407799 (+ 372us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:36.407809 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.408988 (+ 1179us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:36.408992 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:36.409785 (+ 793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:36.409797 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.409936 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:36.410568 (+ 632us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:36.410585 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:36.413091 (+ 2506us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":329,"thread_start_us":211,"threads_started":1}
W20251027 14:52:36.413537 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:36.406464 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:39133 (local address 127.0.104.62:41623)
1027 14:52:36.406724 (+ 260us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:36.406730 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:36.406748 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:52:36.406847 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:36.406853 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:36.406919 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:36.407017 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:36.407940 (+ 923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.408860 (+ 920us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:36.409913 (+ 1053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.410352 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:36.413310 (+ 2958us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:39133: BlockingRecv error: recv got EOF from 127.0.104.1:39133 (error 108)
Metrics: {"server-negotiator.queue_time_us":140}
Oct 27 14:52:36 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:36 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:36.871934 10409 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:36.864231 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:43945)
1027 14:52:36.864754 (+ 523us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:36.864774 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:36.864899 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:36.865261 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:36.865266 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:36.865661 (+ 395us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:36.866089 (+ 428us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:36.866100 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.867546 (+ 1446us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:36.867551 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:36.868249 (+ 698us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:36.868260 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.868399 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:36.869056 (+ 657us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:36.869074 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:36.871723 (+ 2649us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":410,"thread_start_us":296,"threads_started":1}
W20251027 14:52:36.872197 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:36.864403 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:43945 (local address 127.0.104.62:41623)
1027 14:52:36.864593 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:36.864598 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:36.864849 (+ 251us) server_negotiation.cc:408] Connection header received
1027 14:52:36.865043 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:36.865048 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:36.865114 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:36.865230 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:36.866215 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.867431 (+ 1216us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:36.868397 (+ 966us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:36.868794 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:36.871964 (+ 3170us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:43945: BlockingRecv error: recv got EOF from 127.0.104.2:43945 (error 108)
Metrics: {"server-negotiator.queue_time_us":75}
Oct 27 14:52:37 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:37 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.3@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:37.290896 10410 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:37.283316 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.3:47005)
1027 14:52:37.283834 (+ 518us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:37.283850 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:37.283981 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:37.284285 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:37.284290 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:37.284684 (+ 394us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:37.285078 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:37.285089 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.286451 (+ 1362us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:37.286458 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:37.287180 (+ 722us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:37.287191 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.287326 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:37.287984 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:37.288002 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:37.290680 (+ 2678us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":430,"thread_start_us":276,"threads_started":1}
W20251027 14:52:37.291111 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:37.283412 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:47005 (local address 127.0.104.62:41623)
1027 14:52:37.283626 (+ 214us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:37.283633 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:37.283931 (+ 298us) server_negotiation.cc:408] Connection header received
1027 14:52:37.284094 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:37.284100 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:37.284161 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:37.284240 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:37.285331 (+ 1091us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.286307 (+ 976us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:37.287325 (+ 1018us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.287755 (+ 430us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:37.290911 (+ 3156us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.3:47005: BlockingRecv error: recv got EOF from 127.0.104.3:47005 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
Oct 27 14:52:37 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:37 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.1@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:37.421756 10411 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:37.414517 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.1:45557)
1027 14:52:37.414985 (+ 468us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:37.414999 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:37.415119 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:37.415426 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:37.415431 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:37.415855 (+ 424us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:37.416224 (+ 369us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:37.416234 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.417416 (+ 1182us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:37.417421 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:37.418163 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:37.418174 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.418303 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:37.418921 (+ 618us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:37.418939 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:37.421539 (+ 2600us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":377,"thread_start_us":246,"threads_started":1}
W20251027 14:52:37.421968 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:37.414641 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:45557 (local address 127.0.104.62:41623)
1027 14:52:37.414809 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:37.414815 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:37.415071 (+ 256us) server_negotiation.cc:408] Connection header received
1027 14:52:37.415238 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:37.415242 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:37.415301 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:37.415386 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:37.416413 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.417282 (+ 869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:37.418288 (+ 1006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.418741 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:37.421772 (+ 3031us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.1:45557: BlockingRecv error: recv got EOF from 127.0.104.1:45557 (error 108)
Metrics: {"server-negotiator.queue_time_us":50}
Oct 27 14:52:37 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
Oct 27 14:52:37 dist-test-slave-hxxt krb5kdc[9356](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: LOOKING_UP_SERVER: authtime 0, oryx/127.0.104.2@KRBTEST.COM for oryx/127.0.104.62@KRBTEST.COM, Server not found in Kerberos database
I20251027 14:52:37.884191 10412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:37.872984 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41623 (local address 127.0.104.2:53477)
1027 14:52:37.873495 (+ 511us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:37.873511 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:37.873649 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:37.873943 (+ 294us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:37.873947 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:37.874377 (+ 430us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:37.874750 (+ 373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:37.874760 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.876127 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:37.876132 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:37.877004 (+ 872us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:37.877018 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.877147 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:37.881301 (+ 4154us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:37.881321 (+ 20us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:37.883896 (+ 2575us) negotiation.cc:327] Negotiation complete: Not authorized: Client connection negotiation failed: client connection to 127.0.104.62:41623: Server oryx/127.0.104.62@KRBTEST.COM not found in Kerberos database
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":282,"threads_started":1}
W20251027 14:52:37.884368 10103 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:37.873132 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:53477 (local address 127.0.104.62:41623)
1027 14:52:37.873365 (+ 233us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:37.873370 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:37.873600 (+ 230us) server_negotiation.cc:408] Connection header received
1027 14:52:37.873750 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:37.873754 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:37.873831 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:37.873912 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:37.874898 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.875992 (+ 1094us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:37.877133 (+ 1141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:37.877575 (+ 442us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:37.884153 (+ 6578us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.104.2:53477: BlockingRecv error: recv got EOF from 127.0.104.2:53477 (error 108)
Metrics: {"server-negotiator.queue_time_us":99}
I20251027 14:52:38.153659 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9954
I20251027 14:52:38.164325 10414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:38.154741 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46598 (local address 127.0.104.1:44087)
1027 14:52:38.155257 (+ 516us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:38.155265 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:38.155297 (+ 32us) server_negotiation.cc:408] Connection header received
1027 14:52:38.155360 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:38.155367 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:38.155599 (+ 232us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:38.155789 (+ 190us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:38.157354 (+ 1565us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.158434 (+ 1080us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:38.159477 (+ 1043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.159829 (+ 352us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:38.161584 (+ 1755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:38.161616 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:38.161632 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:38.161658 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:38.163812 (+ 2154us) server_negotiation.cc:395] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.104.1@KRBTEST.COM not found in keytab (ticket kvno 2)
1027 14:52:38.164104 (+ 292us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:46598: Request ticket server kudu/127.0.104.1@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":260,"threads_started":1}
W20251027 14:52:38.164441 10414 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:46598: Request ticket server kudu/127.0.104.1@KRBTEST.COM not found in keytab (ticket kvno 2)
W20251027 14:52:38.164533 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9954: Not authorized: Client connection negotiation failed: client connection to 127.0.104.1:44087: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.104.1@KRBTEST.COM not found in keytab (ticket kvno 2)
I20251027 14:52:38.164707 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9954
I20251027 14:52:38.178717 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10105
I20251027 14:52:38.189914 10415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:38.179694 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38414 (local address 127.0.104.2:39973)
1027 14:52:38.180282 (+ 588us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:38.180289 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:38.180324 (+ 35us) server_negotiation.cc:408] Connection header received
1027 14:52:38.180409 (+ 85us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:38.180420 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:38.180658 (+ 238us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:38.180861 (+ 203us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:38.182322 (+ 1461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.183541 (+ 1219us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:38.184773 (+ 1232us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.185122 (+ 349us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:38.186921 (+ 1799us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:38.186985 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:38.187000 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:38.187038 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:38.189380 (+ 2342us) server_negotiation.cc:395] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.104.2@KRBTEST.COM not found in keytab (ticket kvno 2)
1027 14:52:38.189681 (+ 301us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:38414: Request ticket server kudu/127.0.104.2@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":453,"thread_start_us":314,"threads_started":1}
W20251027 14:52:38.190011 10415 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:38414: Request ticket server kudu/127.0.104.2@KRBTEST.COM not found in keytab (ticket kvno 2)
W20251027 14:52:38.190093 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10105: Not authorized: Client connection negotiation failed: client connection to 127.0.104.2:39973: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.104.2@KRBTEST.COM not found in keytab (ticket kvno 2)
I20251027 14:52:38.190248 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10105
I20251027 14:52:38.203581 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10236
I20251027 14:52:38.213991 10416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:38.204506 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56028 (local address 127.0.104.3:36429)
1027 14:52:38.204993 (+ 487us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:38.205001 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:38.205028 (+ 27us) server_negotiation.cc:408] Connection header received
1027 14:52:38.205093 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:38.205099 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:38.205314 (+ 215us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:38.205508 (+ 194us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:38.206878 (+ 1370us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.207956 (+ 1078us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:38.209060 (+ 1104us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.209496 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:38.211117 (+ 1621us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:38.211148 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:38.211162 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:38.211187 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:38.213451 (+ 2264us) server_negotiation.cc:395] Sending RPC error: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.104.3@KRBTEST.COM not found in keytab (ticket kvno 2)
1027 14:52:38.213734 (+ 283us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:56028: Request ticket server kudu/127.0.104.3@KRBTEST.COM not found in keytab (ticket kvno 2)
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":227,"threads_started":1}
W20251027 14:52:38.214061 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10236: Not authorized: Client connection negotiation failed: client connection to 127.0.104.3:36429: FATAL_UNAUTHORIZED: Not authorized: Request ticket server kudu/127.0.104.3@KRBTEST.COM not found in keytab (ticket kvno 2)
W20251027 14:52:38.214088 10416 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:56028: Request ticket server kudu/127.0.104.3@KRBTEST.COM not found in keytab (ticket kvno 2)
I20251027 14:52:38.214195 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10236
I20251027 14:52:38.227447 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 9885
I20251027 14:52:38.228634 9947 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:38.341430 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 9885
2025-10-27T14:52:38Z chronyd exiting
[ OK ] SecurityITest.TestMismatchingPrincipals (21869 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationInsecureCluster
2025-10-27T14:52:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:38Z Disabled control of system clock
I20251027 14:52:38.415061 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41069
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41069
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {}
W20251027 14:52:38.735527 10421 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:38.735826 10421 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:38.735905 10421 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:38.743968 10421 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:38.744063 10421 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:38.744117 10421 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:38.744163 10421 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:38.744227 10421 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:38.753986 10421 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41069
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:38.755491 10421 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:38.757263 10421 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:38.764889 10426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:38.764983 10429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:38.765013 10427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:38.765646 10421 server_base.cc:1047] running on GCE node
I20251027 14:52:38.766333 10421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:38.767669 10421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:38.768883 10421 hybrid_clock.cc:648] HybridClock initialized: now 1761576758768815 us; error 34 us; skew 500 ppm
I20251027 14:52:38.769207 10421 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:38.771252 10421 webserver.cc:492] Webserver started at http://127.0.104.62:42649/ using document root <none> and password file <none>
I20251027 14:52:38.771910 10421 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:38.772007 10421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:38.772296 10421 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:38.774855 10421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "e315db250ea1437db466069ab4d777b1"
format_stamp: "Formatted at 2025-10-27 14:52:38 on dist-test-slave-hxxt"
server_key: "8eda20e20e17c37635f7be06bb7ddc25"
server_key_iv: "b65c542923a7639255f462d41b8e26e2"
server_key_version: "encryptionkey@0"
I20251027 14:52:38.775578 10421 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "e315db250ea1437db466069ab4d777b1"
format_stamp: "Formatted at 2025-10-27 14:52:38 on dist-test-slave-hxxt"
server_key: "8eda20e20e17c37635f7be06bb7ddc25"
server_key_iv: "b65c542923a7639255f462d41b8e26e2"
server_key_version: "encryptionkey@0"
I20251027 14:52:38.780905 10421 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20251027 14:52:38.784453 10435 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:38.785857 10421 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:38.786003 10421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "e315db250ea1437db466069ab4d777b1"
format_stamp: "Formatted at 2025-10-27 14:52:38 on dist-test-slave-hxxt"
server_key: "8eda20e20e17c37635f7be06bb7ddc25"
server_key_iv: "b65c542923a7639255f462d41b8e26e2"
server_key_version: "encryptionkey@0"
I20251027 14:52:38.786185 10421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:38.803887 10421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:38.804795 10421 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:38.805037 10421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:38.822935 10421 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41069
I20251027 14:52:38.822970 10486 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41069 every 8 connection(s)
I20251027 14:52:38.824455 10421 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:52:38.824867 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10421
I20251027 14:52:38.825062 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:52:38.825735 416 external_mini_cluster.cc:1468] Setting key a4f00ac8243de95c1fdd942c9157f60f
I20251027 14:52:38.829433 10487 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:38.838729 10490 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:38.827862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57082 (local address 127.0.104.62:41069)
1027 14:52:38.828810 (+ 948us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:38.828828 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:38.828876 (+ 48us) server_negotiation.cc:408] Connection header received
1027 14:52:38.830222 (+ 1346us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:38.830246 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:38.830765 (+ 519us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:38.831182 (+ 417us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:38.832190 (+ 1008us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.833618 (+ 1428us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:38.835254 (+ 1636us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:38.835762 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:38.836271 (+ 509us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:38.836359 (+ 88us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:38.836442 (+ 83us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:38.837000 (+ 558us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:38.837046 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:38.837066 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:38.837163 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:38.837429 (+ 266us) server_negotiation.cc:300] Negotiation successful
1027 14:52:38.837763 (+ 334us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20251027 14:52:38.841126 10487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1: Bootstrap starting.
I20251027 14:52:38.844343 10487 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:38.845526 10487 log.cc:826] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:38.848470 10487 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1: No bootstrap required, opened a new log
I20251027 14:52:38.853389 10487 raft_consensus.cc:359] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e315db250ea1437db466069ab4d777b1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41069 } }
I20251027 14:52:38.853734 10487 raft_consensus.cc:385] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:38.853827 10487 raft_consensus.cc:740] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e315db250ea1437db466069ab4d777b1, State: Initialized, Role: FOLLOWER
I20251027 14:52:38.854462 10487 consensus_queue.cc:260] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e315db250ea1437db466069ab4d777b1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41069 } }
I20251027 14:52:38.854702 10487 raft_consensus.cc:399] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:52:38.854831 10487 raft_consensus.cc:493] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:52:38.855010 10487 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:38.857260 10487 raft_consensus.cc:515] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e315db250ea1437db466069ab4d777b1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41069 } }
I20251027 14:52:38.857911 10487 leader_election.cc:304] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e315db250ea1437db466069ab4d777b1; no voters:
I20251027 14:52:38.858573 10487 leader_election.cc:290] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:52:38.858696 10492 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:38.859764 10492 raft_consensus.cc:697] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [term 1 LEADER]: Becoming Leader. State: Replica: e315db250ea1437db466069ab4d777b1, State: Running, Role: LEADER
I20251027 14:52:38.860347 10492 consensus_queue.cc:237] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e315db250ea1437db466069ab4d777b1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41069 } }
I20251027 14:52:38.860813 10487 sys_catalog.cc:565] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:52:38.863124 10493 sys_catalog.cc:455] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e315db250ea1437db466069ab4d777b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e315db250ea1437db466069ab4d777b1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41069 } } }
I20251027 14:52:38.863507 10493 sys_catalog.cc:458] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:38.863523 10494 sys_catalog.cc:455] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e315db250ea1437db466069ab4d777b1. Latest consensus state: current_term: 1 leader_uuid: "e315db250ea1437db466069ab4d777b1" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e315db250ea1437db466069ab4d777b1" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41069 } } }
I20251027 14:52:38.864058 10494 sys_catalog.cc:458] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:38.864615 10499 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:38.872185 10499 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:38.882051 10499 catalog_manager.cc:1357] Generated new cluster ID: 4591e340794049bc88580390841a91d2
I20251027 14:52:38.882181 10499 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:38.899775 10499 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:52:38.900974 10499 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:38.918152 10499 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e315db250ea1437db466069ab4d777b1: Generated new TSK 0
I20251027 14:52:38.919013 10499 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:52:38.925321 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41069
--builtin_ntp_servers=127.0.104.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
W20251027 14:52:39.252167 10511 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:39.252507 10511 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:39.252625 10511 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:39.260586 10511 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:39.260674 10511 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:39.260864 10511 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:39.270424 10511 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:39.271941 10511 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:39.273703 10511 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:39.282186 10516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:39.282217 10517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:39.283092 10519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:39.283396 10511 server_base.cc:1047] running on GCE node
I20251027 14:52:39.283946 10511 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:39.284819 10511 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:39.286031 10511 hybrid_clock.cc:648] HybridClock initialized: now 1761576759285978 us; error 28 us; skew 500 ppm
I20251027 14:52:39.286348 10511 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:39.288427 10511 webserver.cc:492] Webserver started at http://127.0.104.1:35841/ using document root <none> and password file <none>
I20251027 14:52:39.289122 10511 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:39.289259 10511 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:39.289592 10511 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:39.292160 10511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "2deadce2f78c45c0ab11ec5d658d8132"
format_stamp: "Formatted at 2025-10-27 14:52:39 on dist-test-slave-hxxt"
server_key: "e4e94a3d54da32b71812fc62ae05a882"
server_key_iv: "71045e49361d5938608f3517926c127c"
server_key_version: "encryptionkey@0"
I20251027 14:52:39.292898 10511 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "2deadce2f78c45c0ab11ec5d658d8132"
format_stamp: "Formatted at 2025-10-27 14:52:39 on dist-test-slave-hxxt"
server_key: "e4e94a3d54da32b71812fc62ae05a882"
server_key_iv: "71045e49361d5938608f3517926c127c"
server_key_version: "encryptionkey@0"
I20251027 14:52:39.297909 10511 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:52:39.301235 10525 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:39.302520 10511 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251027 14:52:39.302663 10511 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "2deadce2f78c45c0ab11ec5d658d8132"
format_stamp: "Formatted at 2025-10-27 14:52:39 on dist-test-slave-hxxt"
server_key: "e4e94a3d54da32b71812fc62ae05a882"
server_key_iv: "71045e49361d5938608f3517926c127c"
server_key_version: "encryptionkey@0"
I20251027 14:52:39.302845 10511 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:39.320405 10511 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:39.321219 10511 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:39.321480 10511 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:39.322351 10511 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:39.323918 10511 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:39.324000 10511 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:39.324119 10511 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:39.324173 10511 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:39.354111 10511 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33011
I20251027 14:52:39.354151 10637 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33011 every 8 connection(s)
I20251027 14:52:39.355420 10511 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:39.359208 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10511
I20251027 14:52:39.359414 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:52:39.359968 416 external_mini_cluster.cc:1468] Setting key cec360177ef0189d3238d648842f82a8
I20251027 14:52:39.362978 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41069
--builtin_ntp_servers=127.0.104.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251027 14:52:39.367772 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:39.358054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:44277 (local address 127.0.104.62:41069)
1027 14:52:39.358457 (+ 403us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:39.358462 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:39.359937 (+ 1475us) server_negotiation.cc:408] Connection header received
1027 14:52:39.361605 (+ 1668us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:39.361611 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:39.361682 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:39.361795 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:39.363388 (+ 1593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.364588 (+ 1200us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:39.365918 (+ 1330us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.366280 (+ 362us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:39.367033 (+ 753us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:39.367066 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:39.367070 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:39.367085 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:39.367118 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:39.367130 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:39.367228 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:39.367464 (+ 236us) server_negotiation.cc:300] Negotiation successful
1027 14:52:39.367548 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":302,"thread_start_us":212,"threads_started":1}
I20251027 14:52:39.369030 10641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:39.358688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41069 (local address 127.0.104.1:44277)
1027 14:52:39.359762 (+ 1074us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:39.359806 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:39.361341 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:39.362016 (+ 675us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:39.362029 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:39.362168 (+ 139us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:39.363183 (+ 1015us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:39.363223 (+ 40us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.364743 (+ 1520us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:39.364749 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:39.365777 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:39.365791 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.366036 (+ 245us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:39.366752 (+ 716us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:52:39.366755 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:39.366791 (+ 36us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:52:39.366871 (+ 80us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:39.367235 (+ 364us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:39.367244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:39.367251 (+ 7us) client_negotiation.cc:769] Sending connection context
1027 14:52:39.367467 (+ 216us) client_negotiation.cc:240] Negotiation successful
1027 14:52:39.367810 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":552,"thread_start_us":342,"threads_started":1}
I20251027 14:52:39.370523 10638 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41069
I20251027 14:52:39.370850 10638 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:39.371611 10638 heartbeater.cc:507] Master 127.0.104.62:41069 requested a full tablet report, sending...
I20251027 14:52:39.373631 10452 ts_manager.cc:194] Registered new tserver with Master: 2deadce2f78c45c0ab11ec5d658d8132 (127.0.104.1:33011)
I20251027 14:52:39.375087 10452 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:44277
W20251027 14:52:39.687387 10642 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:39.687696 10642 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:39.687815 10642 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:39.695928 10642 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:39.696030 10642 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:39.696185 10642 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:39.705891 10642 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:39.707464 10642 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:39.709283 10642 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:39.718398 10647 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:39.718952 10648 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:39.720885 10650 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:39.720973 10642 server_base.cc:1047] running on GCE node
I20251027 14:52:39.721642 10642 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:39.722592 10642 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:39.723809 10642 hybrid_clock.cc:648] HybridClock initialized: now 1761576759723738 us; error 51 us; skew 500 ppm
I20251027 14:52:39.724093 10642 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:39.726279 10642 webserver.cc:492] Webserver started at http://127.0.104.2:35353/ using document root <none> and password file <none>
I20251027 14:52:39.727133 10642 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:39.727252 10642 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:39.727602 10642 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:39.730196 10642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "0c32df43e5884673a0b4166f98f70105"
format_stamp: "Formatted at 2025-10-27 14:52:39 on dist-test-slave-hxxt"
server_key: "eb2229367f07a067a65fd6760b525da5"
server_key_iv: "efb7d342d586b76b04b40d3db3cd5c09"
server_key_version: "encryptionkey@0"
I20251027 14:52:39.730960 10642 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "0c32df43e5884673a0b4166f98f70105"
format_stamp: "Formatted at 2025-10-27 14:52:39 on dist-test-slave-hxxt"
server_key: "eb2229367f07a067a65fd6760b525da5"
server_key_iv: "efb7d342d586b76b04b40d3db3cd5c09"
server_key_version: "encryptionkey@0"
I20251027 14:52:39.736271 10642 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:52:39.739642 10656 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:39.741073 10642 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:52:39.741236 10642 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "0c32df43e5884673a0b4166f98f70105"
format_stamp: "Formatted at 2025-10-27 14:52:39 on dist-test-slave-hxxt"
server_key: "eb2229367f07a067a65fd6760b525da5"
server_key_iv: "efb7d342d586b76b04b40d3db3cd5c09"
server_key_version: "encryptionkey@0"
I20251027 14:52:39.741417 10642 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:39.760870 10642 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:39.761763 10642 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:39.762042 10642 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:39.763062 10642 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:39.764753 10642 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:39.764853 10642 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:39.764988 10642 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:39.765041 10642 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:39.794706 10642 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:45231
I20251027 14:52:39.794741 10768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:45231 every 8 connection(s)
I20251027 14:52:39.796025 10642 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:39.796563 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10642
I20251027 14:52:39.796777 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:52:39.797230 416 external_mini_cluster.cc:1468] Setting key c108031c552d8a4d8c75fc5c2178778f
I20251027 14:52:39.800665 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41069
--builtin_ntp_servers=127.0.104.20:35667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {}
I20251027 14:52:39.808168 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:39.798883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:60665 (local address 127.0.104.62:41069)
1027 14:52:39.799059 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:39.799065 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:39.801142 (+ 2077us) server_negotiation.cc:408] Connection header received
1027 14:52:39.802701 (+ 1559us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:39.802707 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:39.802793 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:39.802884 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:39.804101 (+ 1217us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.805017 (+ 916us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:39.806090 (+ 1073us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.806521 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:39.807302 (+ 781us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:39.807337 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:39.807340 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:39.807356 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:39.807384 (+ 28us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:39.807396 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:39.807473 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:39.807747 (+ 274us) server_negotiation.cc:300] Negotiation successful
1027 14:52:39.807863 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:52:39.809238 10771 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:39.799744 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41069 (local address 127.0.104.2:60665)
1027 14:52:39.800997 (+ 1253us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:39.801034 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:39.802420 (+ 1386us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:39.803109 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:39.803118 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:39.803224 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:39.803940 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:39.803958 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.805172 (+ 1214us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:39.805177 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:39.805961 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:39.805972 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:39.806229 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:39.807031 (+ 802us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:52:39.807033 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:39.807062 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:52:39.807143 (+ 81us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:39.807485 (+ 342us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:39.807495 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:39.807504 (+ 9us) client_negotiation.cc:769] Sending connection context
1027 14:52:39.807724 (+ 220us) client_negotiation.cc:240] Negotiation successful
1027 14:52:39.808078 (+ 354us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":689,"thread_start_us":397,"threads_started":1}
I20251027 14:52:39.810595 10769 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41069
I20251027 14:52:39.810979 10769 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:39.811904 10769 heartbeater.cc:507] Master 127.0.104.62:41069 requested a full tablet report, sending...
I20251027 14:52:39.813468 10452 ts_manager.cc:194] Registered new tserver with Master: 0c32df43e5884673a0b4166f98f70105 (127.0.104.2:45231)
I20251027 14:52:39.814193 10452 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:60665
W20251027 14:52:40.123355 10772 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:40.123639 10772 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:40.123756 10772 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:40.131745 10772 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:40.131842 10772 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:40.132009 10772 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:40.141466 10772 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:41069
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:40.143033 10772 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:40.144785 10772 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:40.153612 10777 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:40.153589 10778 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:40.154188 10780 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:40.154861 10772 server_base.cc:1047] running on GCE node
I20251027 14:52:40.155438 10772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:40.156286 10772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:40.157500 10772 hybrid_clock.cc:648] HybridClock initialized: now 1761576760157431 us; error 41 us; skew 500 ppm
I20251027 14:52:40.157802 10772 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:40.159802 10772 webserver.cc:492] Webserver started at http://127.0.104.3:40121/ using document root <none> and password file <none>
I20251027 14:52:40.160528 10772 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:40.160636 10772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:40.160919 10772 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:40.163524 10772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "fef025698808456cbbe549d191c736c6"
format_stamp: "Formatted at 2025-10-27 14:52:40 on dist-test-slave-hxxt"
server_key: "3b544cd8c4ecb3662e1b83f7ba66ac08"
server_key_iv: "87bb3b892e9461bb75dba4042eee2974"
server_key_version: "encryptionkey@0"
I20251027 14:52:40.164225 10772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "fef025698808456cbbe549d191c736c6"
format_stamp: "Formatted at 2025-10-27 14:52:40 on dist-test-slave-hxxt"
server_key: "3b544cd8c4ecb3662e1b83f7ba66ac08"
server_key_iv: "87bb3b892e9461bb75dba4042eee2974"
server_key_version: "encryptionkey@0"
I20251027 14:52:40.169735 10772 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:52:40.173249 10786 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:40.174695 10772 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:40.174860 10772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "fef025698808456cbbe549d191c736c6"
format_stamp: "Formatted at 2025-10-27 14:52:40 on dist-test-slave-hxxt"
server_key: "3b544cd8c4ecb3662e1b83f7ba66ac08"
server_key_iv: "87bb3b892e9461bb75dba4042eee2974"
server_key_version: "encryptionkey@0"
I20251027 14:52:40.175064 10772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:40.199146 10772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:40.200026 10772 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:40.200305 10772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:40.201319 10772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:40.202989 10772 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:40.203068 10772 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:40.203166 10772 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:40.203219 10772 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:40.233131 10772 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:35253
I20251027 14:52:40.233186 10898 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:35253 every 8 connection(s)
I20251027 14:52:40.234490 10772 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:40.244050 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10772
I20251027 14:52:40.244249 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:52:40.244745 416 external_mini_cluster.cc:1468] Setting key 117e66f2eec6994c0431a9dd904c8622
I20251027 14:52:40.245853 10640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:40.237108 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:55749 (local address 127.0.104.62:41069)
1027 14:52:40.237325 (+ 217us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:40.237330 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:40.238783 (+ 1453us) server_negotiation.cc:408] Connection header received
1027 14:52:40.240326 (+ 1543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:40.240332 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:40.240459 (+ 127us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:40.240548 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:40.241857 (+ 1309us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.242687 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:40.243759 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.244209 (+ 450us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:40.244968 (+ 759us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:40.245010 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:40.245015 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:40.245041 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:40.245071 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:40.245083 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:40.245161 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:40.245458 (+ 297us) server_negotiation.cc:300] Negotiation successful
1027 14:52:40.245548 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251027 14:52:40.247007 10901 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:40.237655 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41069 (local address 127.0.104.3:55749)
1027 14:52:40.238608 (+ 953us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:40.238643 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:40.240019 (+ 1376us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:40.240820 (+ 801us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:40.240830 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:40.240938 (+ 108us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:40.241684 (+ 746us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:40.241703 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.242842 (+ 1139us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:40.242846 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:40.243628 (+ 782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:40.243639 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.243889 (+ 250us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:40.244715 (+ 826us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:52:40.244718 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:40.244746 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:52:40.244811 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:40.245183 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:40.245194 (+ 11us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:40.245201 (+ 7us) client_negotiation.cc:769] Sending connection context
1027 14:52:40.245419 (+ 218us) client_negotiation.cc:240] Negotiation successful
1027 14:52:40.245743 (+ 324us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":318,"threads_started":1}
I20251027 14:52:40.248286 10899 heartbeater.cc:344] Connected to a master server at 127.0.104.62:41069
I20251027 14:52:40.248621 10899 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:40.249351 10899 heartbeater.cc:507] Master 127.0.104.62:41069 requested a full tablet report, sending...
I20251027 14:52:40.250646 10452 ts_manager.cc:194] Registered new tserver with Master: fef025698808456cbbe549d191c736c6 (127.0.104.3:35253)
I20251027 14:52:40.251417 10452 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:55749
I20251027 14:52:40.260097 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251027 14:52:40.266117 10640 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:40.264596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57094 (local address 127.0.104.62:41069)
1027 14:52:40.264750 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:40.264754 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:40.265244 (+ 490us) server_negotiation.cc:408] Connection header received
1027 14:52:40.265373 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:40.265378 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:40.265449 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:40.265540 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:40.265859 (+ 319us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:57094: BlockingRecv error: recv got EOF from 127.0.0.1:57094 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:52:40.266499 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10511
I20251027 14:52:40.273845 10908 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:40.267602 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44634 (local address 127.0.104.1:33011)
1027 14:52:40.268113 (+ 511us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:40.268120 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:40.268146 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:52:40.268202 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:40.268209 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:40.268489 (+ 280us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:40.268649 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:40.269312 (+ 663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.270485 (+ 1173us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:40.271640 (+ 1155us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.271999 (+ 359us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:40.272428 (+ 429us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:40.272491 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:40.272507 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:40.273098 (+ 591us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:40.273143 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:40.273162 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:40.273262 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:40.273459 (+ 197us) server_negotiation.cc:300] Negotiation successful
1027 14:52:40.273599 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":345,"thread_start_us":226,"threads_started":1}
I20251027 14:52:40.275223 10633 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:40.355441 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10511
I20251027 14:52:40.368932 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10642
I20251027 14:52:40.375859 10910 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:40.369787 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39412 (local address 127.0.104.2:45231)
1027 14:52:40.370298 (+ 511us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:40.370306 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:40.370331 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:52:40.370389 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:40.370396 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:40.370608 (+ 212us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:40.370769 (+ 161us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:40.371376 (+ 607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.372548 (+ 1172us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:40.373699 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.374136 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:40.374514 (+ 378us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:40.374578 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:40.374594 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:40.375163 (+ 569us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:40.375204 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:40.375221 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:40.375328 (+ 107us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:40.375463 (+ 135us) server_negotiation.cc:300] Negotiation successful
1027 14:52:40.375605 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":396,"thread_start_us":241,"threads_started":1}
I20251027 14:52:40.377120 10764 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:40.460876 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10642
I20251027 14:52:40.473980 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10772
I20251027 14:52:40.481307 10912 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:40.474814 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56468 (local address 127.0.104.3:35253)
1027 14:52:40.475401 (+ 587us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:40.475409 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:40.475440 (+ 31us) server_negotiation.cc:408] Connection header received
1027 14:52:40.475500 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:40.475506 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:40.475722 (+ 216us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:40.475879 (+ 157us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:40.476516 (+ 637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.477780 (+ 1264us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:40.478915 (+ 1135us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:40.479433 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:40.479767 (+ 334us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:40.479829 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:40.479844 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:40.480530 (+ 686us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:40.480572 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:40.480592 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:40.480684 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:40.480902 (+ 218us) server_negotiation.cc:300] Negotiation successful
1027 14:52:40.481037 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":294,"threads_started":1}
I20251027 14:52:40.482550 10894 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:40.564103 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10772
I20251027 14:52:40.577153 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10421
I20251027 14:52:40.578312 10482 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:40.677873 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10421
2025-10-27T14:52:40Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationInsecureCluster (2331 ms)
[ RUN ] SecurityITest.TestRequireEncryptionInsecureCluster
2025-10-27T14:52:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:40Z Disabled control of system clock
I20251027 14:52:40.745277 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43153
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251027 14:52:41.066737 10918 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:41.067044 10918 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:41.067170 10918 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:41.075253 10918 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:41.075361 10918 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:41.075412 10918 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:41.075477 10918 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:41.075551 10918 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:41.086387 10918 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:41.087980 10918 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:41.089788 10918 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:41.098021 10926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:41.098021 10923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:41.098830 10924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:41.098941 10918 server_base.cc:1047] running on GCE node
I20251027 14:52:41.099750 10918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:41.101264 10918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:41.102481 10918 hybrid_clock.cc:648] HybridClock initialized: now 1761576761102422 us; error 38 us; skew 500 ppm
I20251027 14:52:41.102783 10918 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:41.104971 10918 webserver.cc:492] Webserver started at http://127.0.104.62:36195/ using document root <none> and password file <none>
I20251027 14:52:41.105669 10918 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:41.105764 10918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:41.106055 10918 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:41.108796 10918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "7117faab00c846d2bc166af98e325050"
format_stamp: "Formatted at 2025-10-27 14:52:41 on dist-test-slave-hxxt"
server_key: "5b015d61f950df6d239fa3cebe597ee8"
server_key_iv: "de3f29c2b7ee3df0db61c26836aa7985"
server_key_version: "encryptionkey@0"
I20251027 14:52:41.109567 10918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "7117faab00c846d2bc166af98e325050"
format_stamp: "Formatted at 2025-10-27 14:52:41 on dist-test-slave-hxxt"
server_key: "5b015d61f950df6d239fa3cebe597ee8"
server_key_iv: "de3f29c2b7ee3df0db61c26836aa7985"
server_key_version: "encryptionkey@0"
I20251027 14:52:41.114864 10918 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:52:41.118332 10932 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:41.119625 10918 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:52:41.119777 10918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "7117faab00c846d2bc166af98e325050"
format_stamp: "Formatted at 2025-10-27 14:52:41 on dist-test-slave-hxxt"
server_key: "5b015d61f950df6d239fa3cebe597ee8"
server_key_iv: "de3f29c2b7ee3df0db61c26836aa7985"
server_key_version: "encryptionkey@0"
I20251027 14:52:41.119971 10918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:41.126369 10918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:41.127133 10918 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:41.127379 10918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:41.145324 10918 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43153
I20251027 14:52:41.145334 10983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43153 every 8 connection(s)
I20251027 14:52:41.146742 10918 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:52:41.151084 10984 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:41.155984 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 10918
I20251027 14:52:41.156178 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:52:41.156752 416 external_mini_cluster.cc:1468] Setting key 712b774bd37af54709b589e4947354c2
I20251027 14:52:41.161664 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050: Bootstrap starting.
I20251027 14:52:41.164361 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:41.158715 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50640 (local address 127.0.104.62:43153)
1027 14:52:41.159684 (+ 969us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:41.159695 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:41.159735 (+ 40us) server_negotiation.cc:408] Connection header received
1027 14:52:41.160914 (+ 1179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:41.160943 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:41.161387 (+ 444us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:41.161759 (+ 372us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:41.162843 (+ 1084us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:41.162849 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:41.162862 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:41.162874 (+ 12us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:41.162922 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:41.162937 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:41.163097 (+ 160us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:41.163206 (+ 109us) server_negotiation.cc:300] Negotiation successful
1027 14:52:41.163490 (+ 284us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":364,"threads_started":1}
I20251027 14:52:41.165288 10984 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:41.166332 10984 log.cc:826] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:41.169080 10984 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050: No bootstrap required, opened a new log
I20251027 14:52:41.174116 10984 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7117faab00c846d2bc166af98e325050" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43153 } }
I20251027 14:52:41.174459 10984 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:41.174553 10984 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7117faab00c846d2bc166af98e325050, State: Initialized, Role: FOLLOWER
I20251027 14:52:41.175118 10984 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7117faab00c846d2bc166af98e325050" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43153 } }
I20251027 14:52:41.175310 10984 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:52:41.175421 10984 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:52:41.175580 10984 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:41.177883 10984 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7117faab00c846d2bc166af98e325050" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43153 } }
I20251027 14:52:41.178418 10984 leader_election.cc:304] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7117faab00c846d2bc166af98e325050; no voters:
I20251027 14:52:41.179028 10984 leader_election.cc:290] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:52:41.179144 10989 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:41.179759 10989 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [term 1 LEADER]: Becoming Leader. State: Replica: 7117faab00c846d2bc166af98e325050, State: Running, Role: LEADER
I20251027 14:52:41.180325 10989 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7117faab00c846d2bc166af98e325050" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43153 } }
I20251027 14:52:41.181114 10984 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:52:41.183012 10991 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7117faab00c846d2bc166af98e325050. Latest consensus state: current_term: 1 leader_uuid: "7117faab00c846d2bc166af98e325050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7117faab00c846d2bc166af98e325050" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43153 } } }
I20251027 14:52:41.183359 10991 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:41.183794 10990 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7117faab00c846d2bc166af98e325050" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7117faab00c846d2bc166af98e325050" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43153 } } }
I20251027 14:52:41.184082 10990 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:41.184896 10997 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:41.191016 10997 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:41.199648 10997 catalog_manager.cc:1357] Generated new cluster ID: 799e4f9bdd9c497b8a04a62c525b5807
I20251027 14:52:41.199777 10997 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:41.217347 10997 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:52:41.217723 10997 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:41.232587 10997 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7117faab00c846d2bc166af98e325050: Generated new TSK 0
I20251027 14:52:41.233778 10997 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:52:41.238396 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43153
--builtin_ntp_servers=127.0.104.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
W20251027 14:52:41.562474 11008 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:41.562770 11008 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:41.562863 11008 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:41.570881 11008 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:41.570972 11008 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:41.571098 11008 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:41.580693 11008 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:41.582257 11008 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:41.584039 11008 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:41.593598 11014 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:41.593607 11013 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:41.593677 11016 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:41.594703 11008 server_base.cc:1047] running on GCE node
I20251027 14:52:41.595276 11008 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:41.596129 11008 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:41.597374 11008 hybrid_clock.cc:648] HybridClock initialized: now 1761576761597293 us; error 70 us; skew 500 ppm
I20251027 14:52:41.597654 11008 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:41.599804 11008 webserver.cc:492] Webserver started at http://127.0.104.1:39399/ using document root <none> and password file <none>
I20251027 14:52:41.600575 11008 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:41.600696 11008 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:41.601017 11008 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:41.603605 11008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "d2cf5b57f46942f98dc3aba5a588dec5"
format_stamp: "Formatted at 2025-10-27 14:52:41 on dist-test-slave-hxxt"
server_key: "cb79aede592ea88c5b49ce3ddae7dafa"
server_key_iv: "eb9b8e4b6554748bbf74ee00720e7921"
server_key_version: "encryptionkey@0"
I20251027 14:52:41.604365 11008 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "d2cf5b57f46942f98dc3aba5a588dec5"
format_stamp: "Formatted at 2025-10-27 14:52:41 on dist-test-slave-hxxt"
server_key: "cb79aede592ea88c5b49ce3ddae7dafa"
server_key_iv: "eb9b8e4b6554748bbf74ee00720e7921"
server_key_version: "encryptionkey@0"
I20251027 14:52:41.609951 11008 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20251027 14:52:41.613627 11022 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:41.615007 11008 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20251027 14:52:41.615170 11008 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "d2cf5b57f46942f98dc3aba5a588dec5"
format_stamp: "Formatted at 2025-10-27 14:52:41 on dist-test-slave-hxxt"
server_key: "cb79aede592ea88c5b49ce3ddae7dafa"
server_key_iv: "eb9b8e4b6554748bbf74ee00720e7921"
server_key_version: "encryptionkey@0"
I20251027 14:52:41.615368 11008 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:41.622205 11008 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:41.623044 11008 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:41.623448 11008 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:41.624532 11008 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:41.626215 11008 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:41.626289 11008 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:41.626385 11008 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:41.626426 11008 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:41.656524 11008 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34419
I20251027 14:52:41.656555 11134 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34419 every 8 connection(s)
I20251027 14:52:41.657892 11008 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:41.659832 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11008
I20251027 14:52:41.660049 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:52:41.660683 416 external_mini_cluster.cc:1468] Setting key e15384f4730482a67163e417f0cdf0d0
I20251027 14:52:41.663717 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43153
--builtin_ntp_servers=127.0.104.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251027 14:52:41.666407 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:41.660809 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:60179 (local address 127.0.104.62:43153)
1027 14:52:41.661001 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:41.661007 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:41.662723 (+ 1716us) server_negotiation.cc:408] Connection header received
1027 14:52:41.664233 (+ 1510us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:41.664240 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:41.664300 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:41.664428 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:41.665719 (+ 1291us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:41.665724 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:41.665728 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:41.665743 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:41.665777 (+ 34us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:41.665791 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:41.665889 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:41.666121 (+ 232us) server_negotiation.cc:300] Negotiation successful
1027 14:52:41.666174 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:52:41.667351 11137 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:41.661369 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43153 (local address 127.0.104.1:60179)
1027 14:52:41.662561 (+ 1192us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:41.662608 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:41.663950 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:41.664656 (+ 706us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:41.664665 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:41.664764 (+ 99us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:41.665450 (+ 686us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:52:41.665452 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:41.665481 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:52:41.665553 (+ 72us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:41.665884 (+ 331us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:41.665890 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:41.665896 (+ 6us) client_negotiation.cc:769] Sending connection context
1027 14:52:41.666188 (+ 292us) client_negotiation.cc:240] Negotiation successful
1027 14:52:41.666385 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":603,"thread_start_us":388,"threads_started":1}
I20251027 14:52:41.668771 11135 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43153
I20251027 14:52:41.669088 11135 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:41.669639 11135 heartbeater.cc:507] Master 127.0.104.62:43153 requested a full tablet report, sending...
I20251027 14:52:41.671903 10949 ts_manager.cc:194] Registered new tserver with Master: d2cf5b57f46942f98dc3aba5a588dec5 (127.0.104.1:34419)
W20251027 14:52:41.988716 11138 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:41.989008 11138 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:41.989100 11138 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:41.997246 11138 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:41.997334 11138 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:41.997458 11138 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:42.007122 11138 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:42.008705 11138 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:42.010421 11138 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:42.019469 11144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:42.019575 11146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:42.019636 11143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:42.020644 11138 server_base.cc:1047] running on GCE node
I20251027 14:52:42.021237 11138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:42.022150 11138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:42.023381 11138 hybrid_clock.cc:648] HybridClock initialized: now 1761576762023314 us; error 38 us; skew 500 ppm
I20251027 14:52:42.023655 11138 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:42.025849 11138 webserver.cc:492] Webserver started at http://127.0.104.2:43323/ using document root <none> and password file <none>
I20251027 14:52:42.026557 11138 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:42.026680 11138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:42.027001 11138 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:42.030923 11138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "261d55dd62494e68b206ec9ef0c7a95d"
format_stamp: "Formatted at 2025-10-27 14:52:42 on dist-test-slave-hxxt"
server_key: "16c3c56f2f9f6cd2a26f094394ffa235"
server_key_iv: "38682b1b40d9c0c3c4c89bc32720a94d"
server_key_version: "encryptionkey@0"
I20251027 14:52:42.031661 11138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "261d55dd62494e68b206ec9ef0c7a95d"
format_stamp: "Formatted at 2025-10-27 14:52:42 on dist-test-slave-hxxt"
server_key: "16c3c56f2f9f6cd2a26f094394ffa235"
server_key_iv: "38682b1b40d9c0c3c4c89bc32720a94d"
server_key_version: "encryptionkey@0"
I20251027 14:52:42.037016 11138 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:52:42.040544 11152 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:42.041873 11138 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:42.042034 11138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "261d55dd62494e68b206ec9ef0c7a95d"
format_stamp: "Formatted at 2025-10-27 14:52:42 on dist-test-slave-hxxt"
server_key: "16c3c56f2f9f6cd2a26f094394ffa235"
server_key_iv: "38682b1b40d9c0c3c4c89bc32720a94d"
server_key_version: "encryptionkey@0"
I20251027 14:52:42.042254 11138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:42.048657 11138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:42.049460 11138 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:42.049721 11138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:42.050602 11138 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:42.052161 11138 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:42.052243 11138 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:42.052361 11138 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:42.052475 11138 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20251027 14:52:42.081637 11138 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:42325
I20251027 14:52:42.081676 11264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:42325 every 8 connection(s)
I20251027 14:52:42.083279 11138 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:42.087647 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11138
I20251027 14:52:42.087834 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:52:42.088305 416 external_mini_cluster.cc:1468] Setting key 3ce9ef4505b546f888452369bed5881f
I20251027 14:52:42.091136 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43153
--builtin_ntp_servers=127.0.104.20:44141
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_encryption=disabled
--rpc_authentication=disabled with env {}
I20251027 14:52:42.092180 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.086377 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:46847 (local address 127.0.104.62:43153)
1027 14:52:42.086534 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:42.086539 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:42.088086 (+ 1547us) server_negotiation.cc:408] Connection header received
1027 14:52:42.089677 (+ 1591us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:42.089684 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:42.089747 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:42.089834 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:42.091379 (+ 1545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:42.091385 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:42.091390 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:42.091407 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:42.091442 (+ 35us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:42.091456 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:42.091530 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:42.091885 (+ 355us) server_negotiation.cc:300] Negotiation successful
1027 14:52:42.091940 (+ 55us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
I20251027 14:52:42.093530 11267 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.086916 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43153 (local address 127.0.104.2:46847)
1027 14:52:42.087926 (+ 1010us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:42.087966 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:42.089405 (+ 1439us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:42.090105 (+ 700us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:42.090117 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:42.090261 (+ 144us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:42.091002 (+ 741us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:52:42.091006 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:42.091045 (+ 39us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:52:42.091146 (+ 101us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:42.091607 (+ 461us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:42.091615 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:42.091622 (+ 7us) client_negotiation.cc:769] Sending connection context
1027 14:52:42.091942 (+ 320us) client_negotiation.cc:240] Negotiation successful
1027 14:52:42.092215 (+ 273us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":341,"threads_started":1}
I20251027 14:52:42.094946 11265 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43153
I20251027 14:52:42.095292 11265 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:42.095862 11265 heartbeater.cc:507] Master 127.0.104.62:43153 requested a full tablet report, sending...
I20251027 14:52:42.097263 10949 ts_manager.cc:194] Registered new tserver with Master: 261d55dd62494e68b206ec9ef0c7a95d (127.0.104.2:42325)
W20251027 14:52:42.417335 11268 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:42.417629 11268 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:42.417754 11268 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:42.425871 11268 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:42.425966 11268 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:42.426119 11268 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:42.435546 11268 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44141
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:42.437163 11268 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:42.438932 11268 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:42.447495 11274 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:42.448130 11273 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:42.448452 11276 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:42.448694 11268 server_base.cc:1047] running on GCE node
I20251027 14:52:42.449267 11268 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:42.450131 11268 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:42.451349 11268 hybrid_clock.cc:648] HybridClock initialized: now 1761576762451285 us; error 44 us; skew 500 ppm
I20251027 14:52:42.451619 11268 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:52:42.453752 11268 webserver.cc:492] Webserver started at http://127.0.104.3:37627/ using document root <none> and password file <none>
I20251027 14:52:42.454473 11268 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:42.454587 11268 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:42.454957 11268 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:42.457522 11268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "0a66694c83ed49bc8524fffec8b3b3b2"
format_stamp: "Formatted at 2025-10-27 14:52:42 on dist-test-slave-hxxt"
server_key: "421460159b1708cdde1d3af1ca6f0d5a"
server_key_iv: "d400c4c888bbd5447db6c7f1a13ac161"
server_key_version: "encryptionkey@0"
I20251027 14:52:42.458243 11268 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "0a66694c83ed49bc8524fffec8b3b3b2"
format_stamp: "Formatted at 2025-10-27 14:52:42 on dist-test-slave-hxxt"
server_key: "421460159b1708cdde1d3af1ca6f0d5a"
server_key_iv: "d400c4c888bbd5447db6c7f1a13ac161"
server_key_version: "encryptionkey@0"
I20251027 14:52:42.463240 11268 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20251027 14:52:42.466457 11282 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:42.467773 11268 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:52:42.467939 11268 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "0a66694c83ed49bc8524fffec8b3b3b2"
format_stamp: "Formatted at 2025-10-27 14:52:42 on dist-test-slave-hxxt"
server_key: "421460159b1708cdde1d3af1ca6f0d5a"
server_key_iv: "d400c4c888bbd5447db6c7f1a13ac161"
server_key_version: "encryptionkey@0"
I20251027 14:52:42.468132 11268 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:42.474210 11268 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:42.474931 11268 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:42.475193 11268 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:42.476015 11268 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:42.477574 11268 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:42.477649 11268 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:42.477772 11268 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:42.477825 11268 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:42.507187 11268 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:35891
I20251027 14:52:42.507197 11394 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:35891 every 8 connection(s)
I20251027 14:52:42.508580 11268 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:42.514122 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11268
I20251027 14:52:42.514292 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireEncryptionInsecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:52:42.514647 416 external_mini_cluster.cc:1468] Setting key 683e4a3fb13d22e7f43710dbe0452770
I20251027 14:52:42.516768 10987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.511172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:34685 (local address 127.0.104.62:43153)
1027 14:52:42.511350 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:42.511355 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:42.512990 (+ 1635us) server_negotiation.cc:408] Connection header received
1027 14:52:42.514450 (+ 1460us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:42.514457 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:42.514524 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:42.514614 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:42.515988 (+ 1374us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:42.515993 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:42.515997 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:42.516012 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:42.516043 (+ 31us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:42.516054 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:42.516149 (+ 95us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:42.516434 (+ 285us) server_negotiation.cc:300] Negotiation successful
1027 14:52:42.516503 (+ 69us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:52:42.518047 11397 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.511659 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43153 (local address 127.0.104.3:34685)
1027 14:52:42.512789 (+ 1130us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:42.512833 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:42.514144 (+ 1311us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:42.514879 (+ 735us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:42.514888 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:42.514994 (+ 106us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:42.515687 (+ 693us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:52:42.515689 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:42.515718 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:52:42.515791 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:42.516150 (+ 359us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:42.516160 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:42.516168 (+ 8us) client_negotiation.cc:769] Sending connection context
1027 14:52:42.516504 (+ 336us) client_negotiation.cc:240] Negotiation successful
1027 14:52:42.516769 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":535,"thread_start_us":346,"threads_started":1}
I20251027 14:52:42.519315 11395 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43153
I20251027 14:52:42.519645 11395 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:42.520222 11395 heartbeater.cc:507] Master 127.0.104.62:43153 requested a full tablet report, sending...
I20251027 14:52:42.521577 10949 ts_manager.cc:194] Registered new tserver with Master: 0a66694c83ed49bc8524fffec8b3b3b2 (127.0.104.3:35891)
I20251027 14:52:42.529722 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
W20251027 14:52:42.535444 10987 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:52:42.534076 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50652 (local address 127.0.104.62:43153)
1027 14:52:42.534233 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:42.534237 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:42.534678 (+ 441us) server_negotiation.cc:408] Connection header received
1027 14:52:42.534806 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:42.534811 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:42.534860 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:42.534941 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:42.535245 (+ 304us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:50652: BlockingRecv error: recv got EOF from 127.0.0.1:50652 (error 108)
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:52:42.536120 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11008
I20251027 14:52:42.539340 11404 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.536993 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33048 (local address 127.0.104.1:34419)
1027 14:52:42.537470 (+ 477us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:42.537477 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:42.537503 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:52:42.537559 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:42.537565 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:42.537770 (+ 205us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:42.537925 (+ 155us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:42.538733 (+ 808us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:42.538740 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:42.538751 (+ 11us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:42.538760 (+ 9us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:42.538796 (+ 36us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:42.538807 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:42.538945 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:42.538988 (+ 43us) server_negotiation.cc:300] Negotiation successful
1027 14:52:42.539065 (+ 77us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":257,"threads_started":1}
I20251027 14:52:42.540504 11130 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:42.620314 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11008
I20251027 14:52:42.636009 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11138
I20251027 14:52:42.639601 11406 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.636901 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56752 (local address 127.0.104.2:42325)
1027 14:52:42.637397 (+ 496us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:42.637405 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:42.637431 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:52:42.637489 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:42.637495 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:42.637704 (+ 209us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:42.637852 (+ 148us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:42.638894 (+ 1042us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:42.638903 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:42.638915 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:42.638953 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:42.639004 (+ 51us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:42.639020 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:42.639167 (+ 147us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:42.639293 (+ 126us) server_negotiation.cc:300] Negotiation successful
1027 14:52:42.639371 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":259,"threads_started":1}
I20251027 14:52:42.640975 11260 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:42.720516 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11138
I20251027 14:52:42.734179 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11268
I20251027 14:52:42.738047 11408 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:42.735072 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55510 (local address 127.0.104.3:35891)
1027 14:52:42.735682 (+ 610us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:42.735689 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:42.735718 (+ 29us) server_negotiation.cc:408] Connection header received
1027 14:52:42.735777 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:42.735790 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:42.736031 (+ 241us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:42.736198 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:42.737351 (+ 1153us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:42.737359 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:42.737371 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:52:42.737381 (+ 10us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:42.737414 (+ 33us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:52:42.737427 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:42.737591 (+ 164us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:42.737660 (+ 69us) server_negotiation.cc:300] Negotiation successful
1027 14:52:42.737750 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":488,"thread_start_us":352,"threads_started":1}
I20251027 14:52:42.739313 11390 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:42.819854 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11268
I20251027 14:52:42.832557 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 10918
I20251027 14:52:42.833734 10979 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:42.931953 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 10918
2025-10-27T14:52:42Z chronyd exiting
[ OK ] SecurityITest.TestRequireEncryptionInsecureCluster (2254 ms)
[ RUN ] SecurityITest.TestRequireAuthenticationSecureCluster
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:52:43 dist-test-slave-hxxt krb5kdc[11413](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:52:43 dist-test-slave-hxxt krb5kdc[11413](info): set up 2 sockets
Oct 27 14:52:43 dist-test-slave-hxxt krb5kdc[11413](info): commencing operation
krb5kdc: starting...
W20251027 14:52:44.821116 416 mini_kdc.cc:121] Time spent starting KDC: real 1.852s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:52:44 dist-test-slave-hxxt krb5kdc[11413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:52:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:45.084589 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42807
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42807
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:45.402055 11429 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:45.402387 11429 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:45.402509 11429 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:45.410534 11429 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:52:45.410662 11429 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:45.410750 11429 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:45.410810 11429 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:52:45.410888 11429 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:52:45.421278 11429 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42807
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42807
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:45.422816 11429 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:45.424121 11429 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:45.431764 11435 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:45.431802 11434 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:45.432606 11437 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:45.433095 11429 server_base.cc:1047] running on GCE node
I20251027 14:52:45.433781 11429 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:45.435111 11429 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:45.436305 11429 hybrid_clock.cc:648] HybridClock initialized: now 1761576765436241 us; error 45 us; skew 500 ppm
I20251027 14:52:45.436616 11429 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:45 dist-test-slave-hxxt krb5kdc[11413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576765, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:45.439371 11429 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:52:45.440891 11429 webserver.cc:492] Webserver started at http://127.0.104.62:40895/ using document root <none> and password file <none>
I20251027 14:52:45.441573 11429 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:45.441653 11429 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:45.441982 11429 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:45.444466 11429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "9ed80643254d4393aff639a0a9d13116"
format_stamp: "Formatted at 2025-10-27 14:52:45 on dist-test-slave-hxxt"
server_key: "d25b75b35cc6cac843794f8c46d23dd1"
server_key_iv: "92714f89698bbdaa36c853a6ba8d3d9e"
server_key_version: "encryptionkey@0"
I20251027 14:52:45.445217 11429 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "9ed80643254d4393aff639a0a9d13116"
format_stamp: "Formatted at 2025-10-27 14:52:45 on dist-test-slave-hxxt"
server_key: "d25b75b35cc6cac843794f8c46d23dd1"
server_key_iv: "92714f89698bbdaa36c853a6ba8d3d9e"
server_key_version: "encryptionkey@0"
I20251027 14:52:45.450384 11429 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20251027 14:52:45.453533 11444 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:45.454772 11429 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:45.454916 11429 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "9ed80643254d4393aff639a0a9d13116"
format_stamp: "Formatted at 2025-10-27 14:52:45 on dist-test-slave-hxxt"
server_key: "d25b75b35cc6cac843794f8c46d23dd1"
server_key_iv: "92714f89698bbdaa36c853a6ba8d3d9e"
server_key_version: "encryptionkey@0"
I20251027 14:52:45.455118 11429 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:45.479914 11429 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:45.483456 11429 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:45.483702 11429 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:45.501927 11429 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42807
I20251027 14:52:45.501938 11495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42807 every 8 connection(s)
I20251027 14:52:45.503290 11429 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:52:45.506989 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11429
I20251027 14:52:45.507165 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:52:45.507347 11496 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:45.507761 416 external_mini_cluster.cc:1468] Setting key f8715f9976ece0e2695365a66cf817fb
I20251027 14:52:45.519903 11496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116: Bootstrap starting.
Oct 27 14:52:45 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:45.523415 11496 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:45.524489 11496 log.cc:826] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:45.527642 11496 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116: No bootstrap required, opened a new log
I20251027 14:52:45.528666 11499 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:45.509688 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44380 (local address 127.0.104.62:42807)
1027 14:52:45.510437 (+ 749us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:45.510448 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:45.510487 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:52:45.512083 (+ 1596us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:45.512105 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:45.512543 (+ 438us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:45.512944 (+ 401us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:45.514862 (+ 1918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:45.516162 (+ 1300us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:45.517673 (+ 1511us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:45.518141 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:45.522113 (+ 3972us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:45.522196 (+ 83us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:45.522213 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:45.522248 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:45.525242 (+ 2994us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:45.525854 (+ 612us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:45.525861 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:45.525867 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:45.525956 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:45.526243 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:45.526246 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:45.526248 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:45.526645 (+ 397us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:45.526865 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:45.527290 (+ 425us) server_negotiation.cc:300] Negotiation successful
1027 14:52:45.527634 (+ 344us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":578,"thread_start_us":391,"threads_started":1}
I20251027 14:52:45.532702 11496 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed80643254d4393aff639a0a9d13116" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42807 } }
I20251027 14:52:45.533020 11496 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:45.533129 11496 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ed80643254d4393aff639a0a9d13116, State: Initialized, Role: FOLLOWER
I20251027 14:52:45.533663 11496 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed80643254d4393aff639a0a9d13116" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42807 } }
I20251027 14:52:45.533847 11496 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:52:45.533958 11496 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:52:45.534096 11496 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:45.536175 11496 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed80643254d4393aff639a0a9d13116" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42807 } }
I20251027 14:52:45.536685 11496 leader_election.cc:304] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9ed80643254d4393aff639a0a9d13116; no voters:
I20251027 14:52:45.537268 11496 leader_election.cc:290] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:52:45.537374 11501 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:45.537946 11501 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [term 1 LEADER]: Becoming Leader. State: Replica: 9ed80643254d4393aff639a0a9d13116, State: Running, Role: LEADER
I20251027 14:52:45.538432 11501 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed80643254d4393aff639a0a9d13116" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42807 } }
I20251027 14:52:45.539072 11496 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:52:45.541312 11503 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ed80643254d4393aff639a0a9d13116. Latest consensus state: current_term: 1 leader_uuid: "9ed80643254d4393aff639a0a9d13116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed80643254d4393aff639a0a9d13116" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42807 } } }
I20251027 14:52:45.541606 11503 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:45.541514 11502 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9ed80643254d4393aff639a0a9d13116" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9ed80643254d4393aff639a0a9d13116" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42807 } } }
I20251027 14:52:45.541754 11502 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116 [sys.catalog]: This master's current role is: LEADER
I20251027 14:52:45.549701 11510 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:52:45.554920 11510 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:52:45.564016 11510 catalog_manager.cc:1357] Generated new cluster ID: 8b3db5f257df4862940fe6b3b1f1a3c4
I20251027 14:52:45.564122 11510 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:52:45.577276 11510 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:52:45.578675 11510 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:52:45.604993 11510 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116: Generated new TSK 0
I20251027 14:52:45.606053 11510 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:45.728510 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42807
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:46.054080 11524 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:46.054394 11524 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:46.054522 11524 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:46.062602 11524 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:46.062709 11524 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:46.062893 11524 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:52:46.073288 11524 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:42807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:46.074865 11524 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:46.076207 11524 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:46.085619 11530 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:46.085772 11529 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:46.086166 11532 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:46.086766 11524 server_base.cc:1047] running on GCE node
I20251027 14:52:46.087400 11524 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:46.088363 11524 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:46.089612 11524 hybrid_clock.cc:648] HybridClock initialized: now 1761576766089559 us; error 65 us; skew 500 ppm
I20251027 14:52:46.089926 11524 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:46 dist-test-slave-hxxt krb5kdc[11413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:46.093020 11524 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:52:46.094511 11524 webserver.cc:492] Webserver started at http://127.0.104.1:33661/ using document root <none> and password file <none>
I20251027 14:52:46.095249 11524 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:46.095327 11524 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:46.095646 11524 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:46.098598 11524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "33c8c1730a2c4ae29290e53265bbfe33"
format_stamp: "Formatted at 2025-10-27 14:52:46 on dist-test-slave-hxxt"
server_key: "97bf6cf3aa275e4f0cc916ed380c284d"
server_key_iv: "0d577596fdf092cb276807f765e1288b"
server_key_version: "encryptionkey@0"
I20251027 14:52:46.099754 11524 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "33c8c1730a2c4ae29290e53265bbfe33"
format_stamp: "Formatted at 2025-10-27 14:52:46 on dist-test-slave-hxxt"
server_key: "97bf6cf3aa275e4f0cc916ed380c284d"
server_key_iv: "0d577596fdf092cb276807f765e1288b"
server_key_version: "encryptionkey@0"
I20251027 14:52:46.105464 11524 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251027 14:52:46.109014 11539 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:46.110384 11524 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:52:46.110520 11524 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "33c8c1730a2c4ae29290e53265bbfe33"
format_stamp: "Formatted at 2025-10-27 14:52:46 on dist-test-slave-hxxt"
server_key: "97bf6cf3aa275e4f0cc916ed380c284d"
server_key_iv: "0d577596fdf092cb276807f765e1288b"
server_key_version: "encryptionkey@0"
I20251027 14:52:46.110741 11524 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:46.126116 11524 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:46.129982 11524 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:46.130216 11524 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:46.131099 11524 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:46.132597 11524 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:46.132683 11524 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:46.132781 11524 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:46.132858 11524 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:46.162220 11524 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:38913
I20251027 14:52:46.162266 11651 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:38913 every 8 connection(s)
I20251027 14:52:46.163579 11524 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:52:46.173810 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11524
I20251027 14:52:46.174002 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:52:46.174681 416 external_mini_cluster.cc:1468] Setting key bd9546d9800d746526e33cc712260267
Oct 27 14:52:46 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:46.182117 11654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:46.166324 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:43901 (local address 127.0.104.62:42807)
1027 14:52:46.166747 (+ 423us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:46.166752 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:46.167790 (+ 1038us) server_negotiation.cc:408] Connection header received
1027 14:52:46.169330 (+ 1540us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:46.169336 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:46.169406 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:46.169513 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:46.171175 (+ 1662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.172067 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:46.173177 (+ 1110us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.173518 (+ 341us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:46.176469 (+ 2951us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:46.176501 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:46.176506 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:46.176541 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:46.179070 (+ 2529us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:46.179897 (+ 827us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:46.179902 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:46.179905 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:46.179993 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:46.180335 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:46.180339 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:46.180341 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:46.180624 (+ 283us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:46.180757 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:46.181642 (+ 885us) server_negotiation.cc:300] Negotiation successful
1027 14:52:46.181864 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":326,"thread_start_us":212,"threads_started":1}
I20251027 14:52:46.183384 11655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:46.166932 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42807 (local address 127.0.104.1:43901)
1027 14:52:46.167626 (+ 694us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:46.167662 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:46.169022 (+ 1360us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:46.169751 (+ 729us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:46.169760 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:46.170321 (+ 561us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:46.171000 (+ 679us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:46.171019 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.172204 (+ 1185us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:46.172208 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:46.173050 (+ 842us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:46.173061 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.173271 (+ 210us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:46.173957 (+ 686us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:46.173986 (+ 29us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:46.176226 (+ 2240us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:46.179219 (+ 2993us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:46.179228 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:46.179247 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:46.179759 (+ 512us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:46.180093 (+ 334us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:46.180097 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:46.180100 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:46.180244 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:46.180740 (+ 496us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:46.180747 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:46.181170 (+ 423us) client_negotiation.cc:769] Sending connection context
1027 14:52:46.181525 (+ 355us) client_negotiation.cc:240] Negotiation successful
1027 14:52:46.182033 (+ 508us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":460,"thread_start_us":292,"threads_started":1}
I20251027 14:52:46.185027 11652 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42807
I20251027 14:52:46.185369 11652 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:46.186306 11652 heartbeater.cc:507] Master 127.0.104.62:42807 requested a full tablet report, sending...
I20251027 14:52:46.188721 11461 ts_manager.cc:194] Registered new tserver with Master: 33c8c1730a2c4ae29290e53265bbfe33 (127.0.104.1:38913)
I20251027 14:52:46.190272 11461 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:43901
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:46.266868 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42807
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:46.589483 11660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:46.589787 11660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:46.589890 11660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:46.598013 11660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:46.598124 11660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:46.598270 11660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:52:46.608664 11660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:42807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:46.610158 11660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:46.611428 11660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:46.620364 11666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:46.620396 11665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:46.620927 11668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:46.621878 11660 server_base.cc:1047] running on GCE node
I20251027 14:52:46.622453 11660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:46.623394 11660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:46.624614 11660 hybrid_clock.cc:648] HybridClock initialized: now 1761576766624554 us; error 65 us; skew 500 ppm
I20251027 14:52:46.624913 11660 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:46 dist-test-slave-hxxt krb5kdc[11413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:46.627998 11660 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:52:46.629549 11660 webserver.cc:492] Webserver started at http://127.0.104.2:33043/ using document root <none> and password file <none>
I20251027 14:52:46.630285 11660 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:46.630373 11660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:46.630721 11660 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:46.633329 11660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "a33e3d14c7d846efad99391863e8c111"
format_stamp: "Formatted at 2025-10-27 14:52:46 on dist-test-slave-hxxt"
server_key: "f4421788ac6d64f67d2219b898d1d662"
server_key_iv: "091bc08fb21998fbb5e3712da13db85b"
server_key_version: "encryptionkey@0"
I20251027 14:52:46.634213 11660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "a33e3d14c7d846efad99391863e8c111"
format_stamp: "Formatted at 2025-10-27 14:52:46 on dist-test-slave-hxxt"
server_key: "f4421788ac6d64f67d2219b898d1d662"
server_key_iv: "091bc08fb21998fbb5e3712da13db85b"
server_key_version: "encryptionkey@0"
I20251027 14:52:46.639526 11660 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:52:46.643011 11675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:46.644439 11660 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:52:46.644603 11660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "a33e3d14c7d846efad99391863e8c111"
format_stamp: "Formatted at 2025-10-27 14:52:46 on dist-test-slave-hxxt"
server_key: "f4421788ac6d64f67d2219b898d1d662"
server_key_iv: "091bc08fb21998fbb5e3712da13db85b"
server_key_version: "encryptionkey@0"
I20251027 14:52:46.644838 11660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:46.667356 11660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:46.670982 11660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:46.671222 11660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:46.672120 11660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:46.673563 11660 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:46.673656 11660 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:46.673755 11660 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:46.673832 11660 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:46.703733 11660 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35337
I20251027 14:52:46.703775 11787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35337 every 8 connection(s)
I20251027 14:52:46.705144 11660 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:52:46.708180 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11660
I20251027 14:52:46.708333 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:52:46.708765 416 external_mini_cluster.cc:1468] Setting key de683da286474edc57083392b2fbfc48
Oct 27 14:52:46 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576766, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:46.726156 11790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:46.707765 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:54219 (local address 127.0.104.62:42807)
1027 14:52:46.708191 (+ 426us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:46.708197 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:46.709357 (+ 1160us) server_negotiation.cc:408] Connection header received
1027 14:52:46.711247 (+ 1890us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:46.711251 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:46.711324 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:46.711409 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:46.713701 (+ 2292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.714586 (+ 885us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:46.715949 (+ 1363us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.716284 (+ 335us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:46.719734 (+ 3450us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:46.719796 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:46.719801 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:46.719854 (+ 53us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:46.722912 (+ 3058us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:46.723976 (+ 1064us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:46.723982 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:46.723985 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:46.724074 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:46.724429 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:46.724433 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:46.724436 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:46.724731 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:46.724867 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:46.725599 (+ 732us) server_negotiation.cc:300] Negotiation successful
1027 14:52:46.725843 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":205,"threads_started":1}
I20251027 14:52:46.727044 11791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:46.708359 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42807 (local address 127.0.104.2:54219)
1027 14:52:46.709184 (+ 825us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:46.709233 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:46.710929 (+ 1696us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:46.711699 (+ 770us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:46.711712 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:46.712536 (+ 824us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:46.713511 (+ 975us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:46.713538 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.714748 (+ 1210us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:46.714754 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:46.715819 (+ 1065us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:46.715834 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:46.716097 (+ 263us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:46.716838 (+ 741us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:46.716871 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:46.719458 (+ 2587us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:46.723131 (+ 3673us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:46.723142 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:46.723163 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:46.723844 (+ 681us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:46.724175 (+ 331us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:46.724179 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:46.724182 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:46.724299 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:46.724872 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:46.724879 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:46.725210 (+ 331us) client_negotiation.cc:769] Sending connection context
1027 14:52:46.725492 (+ 282us) client_negotiation.cc:240] Negotiation successful
1027 14:52:46.725902 (+ 410us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":329,"threads_started":1}
I20251027 14:52:46.728482 11788 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42807
I20251027 14:52:46.728832 11788 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:46.729640 11788 heartbeater.cc:507] Master 127.0.104.62:42807 requested a full tablet report, sending...
I20251027 14:52:46.731179 11461 ts_manager.cc:194] Registered new tserver with Master: a33e3d14c7d846efad99391863e8c111 (127.0.104.2:35337)
I20251027 14:52:46.731981 11461 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:54219
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:52:46.801568 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:42807
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42161
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:52:47.127724 11796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:52:47.128046 11796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:52:47.128180 11796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:52:47.136880 11796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:52:47.136987 11796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:52:47.137169 11796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:52:47.147639 11796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42161
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:42807
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:52:47.149367 11796 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:52:47.150677 11796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:52:47.160030 11802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:47.160239 11801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:52:47.160622 11804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:52:47.161525 11796 server_base.cc:1047] running on GCE node
I20251027 14:52:47.162230 11796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:52:47.163165 11796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:52:47.164402 11796 hybrid_clock.cc:648] HybridClock initialized: now 1761576767164288 us; error 68 us; skew 500 ppm
I20251027 14:52:47.164726 11796 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:52:47 dist-test-slave-hxxt krb5kdc[11413](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:52:47.168097 11796 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:52:47.169807 11796 webserver.cc:492] Webserver started at http://127.0.104.3:33453/ using document root <none> and password file <none>
I20251027 14:52:47.170536 11796 fs_manager.cc:362] Metadata directory not provided
I20251027 14:52:47.170634 11796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:52:47.170992 11796 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:52:47.173655 11796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "8abf85d632b740e8b7274816c143e8f4"
format_stamp: "Formatted at 2025-10-27 14:52:47 on dist-test-slave-hxxt"
server_key: "589cf167a78fe286be9c2b899bd8c87b"
server_key_iv: "becda0076144633e8273fe674efe1f19"
server_key_version: "encryptionkey@0"
I20251027 14:52:47.174436 11796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "8abf85d632b740e8b7274816c143e8f4"
format_stamp: "Formatted at 2025-10-27 14:52:47 on dist-test-slave-hxxt"
server_key: "589cf167a78fe286be9c2b899bd8c87b"
server_key_iv: "becda0076144633e8273fe674efe1f19"
server_key_version: "encryptionkey@0"
I20251027 14:52:47.179899 11796 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:52:47.183549 11811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:47.185099 11796 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251027 14:52:47.185271 11796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "8abf85d632b740e8b7274816c143e8f4"
format_stamp: "Formatted at 2025-10-27 14:52:47 on dist-test-slave-hxxt"
server_key: "589cf167a78fe286be9c2b899bd8c87b"
server_key_iv: "becda0076144633e8273fe674efe1f19"
server_key_version: "encryptionkey@0"
I20251027 14:52:47.185484 11796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:52:47.193881 11652 heartbeater.cc:499] Master 127.0.104.62:42807 was elected leader, sending a full tablet report...
I20251027 14:52:47.210963 11796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:52:47.215117 11796 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:52:47.215418 11796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:52:47.216418 11796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:52:47.217900 11796 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:52:47.217981 11796 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:47.218060 11796 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:52:47.218111 11796 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:52:47.249594 11796 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46069
I20251027 14:52:47.249663 11923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46069 every 8 connection(s)
I20251027 14:52:47.250938 11796 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:52:47.253386 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 11796
I20251027 14:52:47.253564 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:52:47.254021 416 external_mini_cluster.cc:1468] Setting key 72b6db4d8da5c8ac94b601a3b1f2e251
Oct 27 14:52:47 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576767, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:52:47.269737 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.253637 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:36819 (local address 127.0.104.62:42807)
1027 14:52:47.254070 (+ 433us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.254076 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.255202 (+ 1126us) server_negotiation.cc:408] Connection header received
1027 14:52:47.256856 (+ 1654us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.256862 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.256947 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.257052 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:47.258772 (+ 1720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.259793 (+ 1021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.260999 (+ 1206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.261318 (+ 319us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.264151 (+ 2833us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:47.264188 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:47.264192 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:47.264222 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:47.266707 (+ 2485us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.267399 (+ 692us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.267403 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.267405 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.267471 (+ 66us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.267858 (+ 387us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.267862 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.267864 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.268143 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:47.268279 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.269123 (+ 844us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.269384 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":204,"threads_started":1}
I20251027 14:52:47.270668 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.254277 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:42807 (local address 127.0.104.3:36819)
1027 14:52:47.255057 (+ 780us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:47.255106 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:47.256491 (+ 1385us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:47.257235 (+ 744us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:47.257244 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:47.257860 (+ 616us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:52:47.258575 (+ 715us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.258596 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.259963 (+ 1367us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.259968 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:47.260852 (+ 884us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.260865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.261085 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.261832 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:52:47.261859 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:52:47.263965 (+ 2106us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:52:47.266849 (+ 2884us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:47.266858 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:47.266878 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:47.267288 (+ 410us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:47.267593 (+ 305us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:52:47.267597 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:52:47.267600 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:52:47.267736 (+ 136us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:52:47.268271 (+ 535us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:52:47.268280 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:52:47.268713 (+ 433us) client_negotiation.cc:769] Sending connection context
1027 14:52:47.269023 (+ 310us) client_negotiation.cc:240] Negotiation successful
1027 14:52:47.269542 (+ 519us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":340,"threads_started":1}
I20251027 14:52:47.272008 11924 heartbeater.cc:344] Connected to a master server at 127.0.104.62:42807
I20251027 14:52:47.272322 11924 heartbeater.cc:461] Registering TS with master...
I20251027 14:52:47.273161 11924 heartbeater.cc:507] Master 127.0.104.62:42807 requested a full tablet report, sending...
I20251027 14:52:47.274578 11461 ts_manager.cc:194] Registered new tserver with Master: 8abf85d632b740e8b7274816c143e8f4 (127.0.104.3:46069)
I20251027 14:52:47.275377 11461 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:36819
I20251027 14:52:47.281574 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:52:47.297320 11926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.286048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44384 (local address 127.0.104.62:42807)
1027 14:52:47.286245 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.286251 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.286555 (+ 304us) server_negotiation.cc:408] Connection header received
1027 14:52:47.286703 (+ 148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.286707 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.286788 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.286865 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:47.288418 (+ 1553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.289294 (+ 876us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.290565 (+ 1271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.291054 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.292659 (+ 1605us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:47.292697 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:47.292701 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:47.292738 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:47.295201 (+ 2463us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.295784 (+ 583us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.295788 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.295790 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.295851 (+ 61us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.296101 (+ 250us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.296104 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.296106 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.296346 (+ 240us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:47.296555 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.296860 (+ 305us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.297039 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:52:47.300936 11461 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44384:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:52:47.305421 11461 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:52:47.337644 11937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.317894 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:46069 (local address 127.0.0.1:37812)
1027 14:52:47.330357 (+ 12463us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:47.330404 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:47.330656 (+ 252us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:47.331486 (+ 830us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:47.331494 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:47.331539 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:47.332404 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.332418 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.334920 (+ 2502us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.334926 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:47.336738 (+ 1812us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.336750 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.336914 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.336988 (+ 74us) client_negotiation.cc:769] Sending connection context
1027 14:52:47.337126 (+ 138us) client_negotiation.cc:240] Negotiation successful
1027 14:52:47.337312 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12235,"thread_start_us":12100,"threads_started":1}
I20251027 14:52:47.338761 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.318168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37812 (local address 127.0.104.3:46069)
1027 14:52:47.329931 (+ 11763us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.329941 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.330855 (+ 914us) server_negotiation.cc:408] Connection header received
1027 14:52:47.330965 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.330974 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.331249 (+ 275us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.332161 (+ 912us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:47.332635 (+ 474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.334678 (+ 2043us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.336942 (+ 2264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.338059 (+ 1117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.338214 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.338339 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.338510 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11601,"thread_start_us":11453,"threads_started":1}
I20251027 14:52:47.339973 11939 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.318698 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:35337 (local address 127.0.0.1:39080)
1027 14:52:47.330666 (+ 11968us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:47.330689 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:47.330842 (+ 153us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:47.332747 (+ 1905us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:47.332753 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:47.332784 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:47.333565 (+ 781us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.333577 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.337322 (+ 3745us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.337327 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:47.339203 (+ 1876us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.339216 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.339397 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.339421 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:52:47.339496 (+ 75us) client_negotiation.cc:240] Negotiation successful
1027 14:52:47.339638 (+ 142us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11838,"thread_start_us":12233,"threads_started":1}
I20251027 14:52:47.341701 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.318994 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39080 (local address 127.0.104.2:35337)
1027 14:52:47.331726 (+ 12732us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.331735 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.332124 (+ 389us) server_negotiation.cc:408] Connection header received
1027 14:52:47.332228 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.332237 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.332528 (+ 291us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.333780 (+ 1252us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:47.334050 (+ 270us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.337074 (+ 3024us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.340311 (+ 3237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.341170 (+ 859us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.341276 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.341387 (+ 111us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.341505 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12578,"thread_start_us":12778,"threads_started":1}
I20251027 14:52:47.342890 11942 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.332590 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:38913 (local address 127.0.0.1:48196)
1027 14:52:47.336650 (+ 4060us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:47.336668 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:47.336790 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:47.337716 (+ 926us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:47.337724 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:47.337754 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:47.338478 (+ 724us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.338492 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.340739 (+ 2247us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.340745 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:47.342234 (+ 1489us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.342250 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.342425 (+ 175us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.342448 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:52:47.342509 (+ 61us) client_negotiation.cc:240] Negotiation successful
1027 14:52:47.342622 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3962,"thread_start_us":3782,"threads_started":1}
I20251027 14:52:47.343676 11859 tablet_service.cc:1505] Processing CreateTablet for tablet c1e74f26f573405f88d5ef78fdba20ce (DEFAULT_TABLE table=test-table [id=1afb0973a80242e0a3e076d384b0450a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:47.344213 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.331301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48196 (local address 127.0.104.1:38913)
1027 14:52:47.336269 (+ 4968us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.336280 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.336774 (+ 494us) server_negotiation.cc:408] Connection header received
1027 14:52:47.336933 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.336942 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.337198 (+ 256us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.337414 (+ 216us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:47.338660 (+ 1246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.340516 (+ 1856us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.342402 (+ 1886us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.343559 (+ 1157us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.343687 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.343817 (+ 130us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.343974 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4825,"thread_start_us":4715,"threads_started":1}
I20251027 14:52:47.345906 11859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e74f26f573405f88d5ef78fdba20ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:47.345904 11723 tablet_service.cc:1505] Processing CreateTablet for tablet c1e74f26f573405f88d5ef78fdba20ce (DEFAULT_TABLE table=test-table [id=1afb0973a80242e0a3e076d384b0450a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:47.347402 11723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e74f26f573405f88d5ef78fdba20ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:47.348408 11587 tablet_service.cc:1505] Processing CreateTablet for tablet c1e74f26f573405f88d5ef78fdba20ce (DEFAULT_TABLE table=test-table [id=1afb0973a80242e0a3e076d384b0450a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:52:47.350072 11587 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c1e74f26f573405f88d5ef78fdba20ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:52:47.356889 11943 tablet_bootstrap.cc:492] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Bootstrap starting.
I20251027 14:52:47.360175 11943 tablet_bootstrap.cc:654] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:47.361044 11944 tablet_bootstrap.cc:492] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Bootstrap starting.
I20251027 14:52:47.361959 11943 log.cc:826] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:47.364018 11946 tablet_bootstrap.cc:492] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Bootstrap starting.
I20251027 14:52:47.365093 11944 tablet_bootstrap.cc:654] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:47.366237 11944 log.cc:826] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:47.367350 11946 tablet_bootstrap.cc:654] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Neither blocks nor log segments found. Creating new log.
I20251027 14:52:47.367686 11943 tablet_bootstrap.cc:492] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: No bootstrap required, opened a new log
I20251027 14:52:47.367981 11943 ts_tablet_manager.cc:1403] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.003s
I20251027 14:52:47.368431 11946 log.cc:826] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Log is configured to *not* fsync() on all Append() calls
I20251027 14:52:47.369365 11944 tablet_bootstrap.cc:492] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: No bootstrap required, opened a new log
I20251027 14:52:47.369629 11944 ts_tablet_manager.cc:1403] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251027 14:52:47.371420 11946 tablet_bootstrap.cc:492] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: No bootstrap required, opened a new log
I20251027 14:52:47.371691 11946 ts_tablet_manager.cc:1403] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.007s
I20251027 14:52:47.374898 11943 raft_consensus.cc:359] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.375289 11943 raft_consensus.cc:385] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:47.375432 11943 raft_consensus.cc:740] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8abf85d632b740e8b7274816c143e8f4, State: Initialized, Role: FOLLOWER
I20251027 14:52:47.376092 11943 consensus_queue.cc:260] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.376320 11946 raft_consensus.cc:359] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.376425 11944 raft_consensus.cc:359] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.376732 11946 raft_consensus.cc:385] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:47.376861 11946 raft_consensus.cc:740] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 33c8c1730a2c4ae29290e53265bbfe33, State: Initialized, Role: FOLLOWER
I20251027 14:52:47.376880 11944 raft_consensus.cc:385] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:52:47.377167 11924 heartbeater.cc:499] Master 127.0.104.62:42807 was elected leader, sending a full tablet report...
I20251027 14:52:47.377429 11946 consensus_queue.cc:260] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.377743 11943 ts_tablet_manager.cc:1434] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Time spent starting tablet: real 0.010s user 0.007s sys 0.004s
I20251027 14:52:47.378062 11944 raft_consensus.cc:740] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a33e3d14c7d846efad99391863e8c111, State: Initialized, Role: FOLLOWER
I20251027 14:52:47.378803 11946 ts_tablet_manager.cc:1434] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Time spent starting tablet: real 0.007s user 0.003s sys 0.003s
I20251027 14:52:47.378708 11944 consensus_queue.cc:260] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.379720 11788 heartbeater.cc:499] Master 127.0.104.62:42807 was elected leader, sending a full tablet report...
I20251027 14:52:47.380254 11944 ts_tablet_manager.cc:1434] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Time spent starting tablet: real 0.011s user 0.009s sys 0.000s
W20251027 14:52:47.415808 11653 tablet.cc:2378] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:52:47.457113 11789 tablet.cc:2378] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:52:47.502561 11925 tablet.cc:2378] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:52:47.667459 11949 raft_consensus.cc:493] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:52:47.667725 11949 raft_consensus.cc:515] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.669518 11949 leader_election.cc:290] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 33c8c1730a2c4ae29290e53265bbfe33 (127.0.104.1:38913), a33e3d14c7d846efad99391863e8c111 (127.0.104.2:35337)
I20251027 14:52:47.675006 11927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.669804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:38913 (local address 127.0.104.3:58115)
1027 14:52:47.670098 (+ 294us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:47.670116 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:47.670258 (+ 142us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:47.670685 (+ 427us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:47.670690 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:47.670775 (+ 85us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:47.671183 (+ 408us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.671195 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.672701 (+ 1506us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.672707 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:47.674289 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.674305 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.674474 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.674501 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:52:47.674565 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:52:47.674688 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":189,"spinlock_wait_cycles":30080}
I20251027 14:52:47.675503 11952 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.669923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:35337 (local address 127.0.104.3:55485)
1027 14:52:47.670584 (+ 661us) negotiation.cc:107] Waiting for socket to connect
1027 14:52:47.670599 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:52:47.670712 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:52:47.671022 (+ 310us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:52:47.671027 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:52:47.671049 (+ 22us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:52:47.671653 (+ 604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.671663 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.673166 (+ 1503us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.673171 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:52:47.674849 (+ 1678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:52:47.674862 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.675035 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.675058 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:52:47.675123 (+ 65us) client_negotiation.cc:240] Negotiation successful
1027 14:52:47.675225 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":394,"threads_started":1}
I20251027 14:52:47.675734 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.669930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:58115 (local address 127.0.104.1:38913)
1027 14:52:47.670101 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.670108 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.670190 (+ 82us) server_negotiation.cc:408] Connection header received
1027 14:52:47.670434 (+ 244us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.670441 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.670533 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.670626 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:47.671323 (+ 697us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.672549 (+ 1226us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.674441 (+ 1892us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.675249 (+ 808us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.675299 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.675378 (+ 79us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.675498 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:52:47.676456 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.669969 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:55485 (local address 127.0.104.2:35337)
1027 14:52:47.670150 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.670156 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.670698 (+ 542us) server_negotiation.cc:408] Connection header received
1027 14:52:47.670840 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.670844 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.670902 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.670970 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:52:47.671819 (+ 849us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.672969 (+ 1150us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.675028 (+ 2059us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.676030 (+ 1002us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.676078 (+ 48us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.676145 (+ 67us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.676258 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251027 14:52:47.676728 11607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e74f26f573405f88d5ef78fdba20ce" candidate_uuid: "8abf85d632b740e8b7274816c143e8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c8c1730a2c4ae29290e53265bbfe33" is_pre_election: true
I20251027 14:52:47.677171 11607 raft_consensus.cc:2468] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8abf85d632b740e8b7274816c143e8f4 in term 0.
I20251027 14:52:47.677208 11743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e74f26f573405f88d5ef78fdba20ce" candidate_uuid: "8abf85d632b740e8b7274816c143e8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33e3d14c7d846efad99391863e8c111" is_pre_election: true
I20251027 14:52:47.677678 11743 raft_consensus.cc:2468] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8abf85d632b740e8b7274816c143e8f4 in term 0.
I20251027 14:52:47.677774 11813 leader_election.cc:304] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c8c1730a2c4ae29290e53265bbfe33, 8abf85d632b740e8b7274816c143e8f4; no voters:
I20251027 14:52:47.678128 11949 raft_consensus.cc:2804] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:52:47.678264 11949 raft_consensus.cc:493] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:52:47.678357 11949 raft_consensus.cc:3060] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:47.680568 11949 raft_consensus.cc:515] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.681103 11949 leader_election.cc:290] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [CANDIDATE]: Term 1 election: Requested vote from peers 33c8c1730a2c4ae29290e53265bbfe33 (127.0.104.1:38913), a33e3d14c7d846efad99391863e8c111 (127.0.104.2:35337)
I20251027 14:52:47.681520 11607 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e74f26f573405f88d5ef78fdba20ce" candidate_uuid: "8abf85d632b740e8b7274816c143e8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "33c8c1730a2c4ae29290e53265bbfe33"
I20251027 14:52:47.681524 11743 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c1e74f26f573405f88d5ef78fdba20ce" candidate_uuid: "8abf85d632b740e8b7274816c143e8f4" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a33e3d14c7d846efad99391863e8c111"
I20251027 14:52:47.681677 11607 raft_consensus.cc:3060] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:47.681672 11743 raft_consensus.cc:3060] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:52:47.684082 11607 raft_consensus.cc:2468] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8abf85d632b740e8b7274816c143e8f4 in term 1.
I20251027 14:52:47.684283 11743 raft_consensus.cc:2468] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8abf85d632b740e8b7274816c143e8f4 in term 1.
I20251027 14:52:47.684528 11813 leader_election.cc:304] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 33c8c1730a2c4ae29290e53265bbfe33, 8abf85d632b740e8b7274816c143e8f4; no voters:
I20251027 14:52:47.684818 11949 raft_consensus.cc:2804] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:52:47.685168 11949 raft_consensus.cc:697] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 1 LEADER]: Becoming Leader. State: Replica: 8abf85d632b740e8b7274816c143e8f4, State: Running, Role: LEADER
I20251027 14:52:47.685599 11949 consensus_queue.cc:237] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } }
I20251027 14:52:47.690367 11461 catalog_manager.cc:5649] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 reported cstate change: term changed from 0 to 1, leader changed from <none> to 8abf85d632b740e8b7274816c143e8f4 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "8abf85d632b740e8b7274816c143e8f4" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "8abf85d632b740e8b7274816c143e8f4" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46069 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 } health_report { overall_health: UNKNOWN } } }
I20251027 14:52:47.707790 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.702808 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37820 (local address 127.0.104.3:46069)
1027 14:52:47.703012 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.703019 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.703038 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:52:47.703239 (+ 201us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.703244 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.703319 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.703447 (+ 128us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:52:47.703998 (+ 551us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.704951 (+ 953us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.706150 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.706534 (+ 384us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.706684 (+ 150us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:52:47.707198 (+ 514us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:52:47.707326 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.707446 (+ 120us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.707556 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:52:47.716156 11743 raft_consensus.cc:1275] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 1 FOLLOWER]: Refusing update from remote peer 8abf85d632b740e8b7274816c143e8f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:47.716197 11607 raft_consensus.cc:1275] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 1 FOLLOWER]: Refusing update from remote peer 8abf85d632b740e8b7274816c143e8f4: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:52:47.717056 11949 consensus_queue.cc:1048] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a33e3d14c7d846efad99391863e8c111" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35337 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:47.717324 11953 consensus_queue.cc:1048] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [LEADER]: Connected to new peer: Peer: permanent_uuid: "33c8c1730a2c4ae29290e53265bbfe33" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:52:47.747668 11458 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44384:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:47.747932 11458 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:44384:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:52:47.751636 11458 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9ed80643254d4393aff639a0a9d13116: Sending DeleteTablet for 3 replicas of tablet c1e74f26f573405f88d5ef78fdba20ce
I20251027 14:52:47.752910 11859 tablet_service.cc:1552] Processing DeleteTablet for tablet c1e74f26f573405f88d5ef78fdba20ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:47 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:37812
I20251027 14:52:47.752974 11587 tablet_service.cc:1552] Processing DeleteTablet for tablet c1e74f26f573405f88d5ef78fdba20ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:47 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:48196
I20251027 14:52:47.753751 11723 tablet_service.cc:1552] Processing DeleteTablet for tablet c1e74f26f573405f88d5ef78fdba20ce with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:52:47 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:39080
I20251027 14:52:47.756646 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11524
I20251027 14:52:47.756897 11968 tablet_replica.cc:333] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: stopping tablet replica
I20251027 14:52:47.757328 11968 raft_consensus.cc:2243] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:47.757820 11968 raft_consensus.cc:2272] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:47.758652 11967 tablet_replica.cc:333] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: stopping tablet replica
I20251027 14:52:47.759169 11967 raft_consensus.cc:2243] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:52:47.759953 11967 raft_consensus.cc:2272] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:47.760669 11966 tablet_replica.cc:333] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: stopping tablet replica
I20251027 14:52:47.761093 11966 raft_consensus.cc:2243] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:52:47.761619 11966 raft_consensus.cc:2272] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:52:47.762538 11967 ts_tablet_manager.cc:1916] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:47.763437 11966 ts_tablet_manager.cc:1916] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:47.765770 11968 ts_tablet_manager.cc:1916] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:52:47.770852 11966 ts_tablet_manager.cc:1929] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:47.771070 11966 log.cc:1199] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/c1e74f26f573405f88d5ef78fdba20ce
I20251027 14:52:47.771622 11966 ts_tablet_manager.cc:1950] T c1e74f26f573405f88d5ef78fdba20ce P 33c8c1730a2c4ae29290e53265bbfe33: Deleting consensus metadata
I20251027 14:52:47.773736 11446 catalog_manager.cc:4985] TS 33c8c1730a2c4ae29290e53265bbfe33 (127.0.104.1:38913): tablet c1e74f26f573405f88d5ef78fdba20ce (table test-table [id=1afb0973a80242e0a3e076d384b0450a]) successfully deleted
Oct 27 14:52:47 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:52:47.778071 11967 ts_tablet_manager.cc:1929] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:47.778234 11967 log.cc:1199] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/c1e74f26f573405f88d5ef78fdba20ce
I20251027 14:52:47.778683 11967 ts_tablet_manager.cc:1950] T c1e74f26f573405f88d5ef78fdba20ce P 8abf85d632b740e8b7274816c143e8f4: Deleting consensus metadata
I20251027 14:52:47.780341 11446 catalog_manager.cc:4985] TS 8abf85d632b740e8b7274816c143e8f4 (127.0.104.3:46069): tablet c1e74f26f573405f88d5ef78fdba20ce (table test-table [id=1afb0973a80242e0a3e076d384b0450a]) successfully deleted
I20251027 14:52:47.780511 11968 ts_tablet_manager.cc:1929] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:52:47.780668 11968 log.cc:1199] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestRequireAuthenticationSecureCluster.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/c1e74f26f573405f88d5ef78fdba20ce
I20251027 14:52:47.781142 11968 ts_tablet_manager.cc:1950] T c1e74f26f573405f88d5ef78fdba20ce P a33e3d14c7d846efad99391863e8c111: Deleting consensus metadata
I20251027 14:52:47.782981 11447 catalog_manager.cc:4985] TS a33e3d14c7d846efad99391863e8c111 (127.0.104.2:35337): tablet c1e74f26f573405f88d5ef78fdba20ce (table test-table [id=1afb0973a80242e0a3e076d384b0450a]) successfully deleted
I20251027 14:52:47.785072 11941 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.758093 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48212 (local address 127.0.104.1:38913)
1027 14:52:47.758390 (+ 297us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.758396 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.768946 (+ 10550us) server_negotiation.cc:408] Connection header received
1027 14:52:47.769226 (+ 280us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.769233 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.769316 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.769388 (+ 72us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:47.771421 (+ 2033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.772873 (+ 1452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.774519 (+ 1646us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.774987 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.779270 (+ 4283us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:47.779327 (+ 57us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:47.779347 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:47.779387 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:47.782449 (+ 3062us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.783243 (+ 794us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.783251 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.783253 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.783329 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.783694 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.783698 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.783700 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.783963 (+ 263us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:47.784169 (+ 206us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.784580 (+ 411us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.784791 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":181}
I20251027 14:52:47.785733 11647 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:47.874053 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11524
I20251027 14:52:47.888566 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11660
Oct 27 14:52:47 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:52:47.905334 11940 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:47.889396 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39092 (local address 127.0.104.2:35337)
1027 14:52:47.889549 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:47.889554 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:47.889571 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:52:47.889628 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:47.889632 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:47.889693 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:47.889774 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:47.891616 (+ 1842us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.892632 (+ 1016us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:47.894235 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:47.894593 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:47.898672 (+ 4079us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:47.898725 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:47.898755 (+ 30us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:47.898801 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:47.902555 (+ 3754us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.903208 (+ 653us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.903218 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.903222 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.903324 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:47.903710 (+ 386us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:47.903715 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:47.903719 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:47.904098 (+ 379us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:47.904439 (+ 341us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:47.904723 (+ 284us) server_negotiation.cc:300] Negotiation successful
1027 14:52:47.904972 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:52:47.906127 11783 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:47.994180 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11660
I20251027 14:52:48.008654 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11796
Oct 27 14:52:48 dist-test-slave-hxxt krb5kdc[11413](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576764, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:52:48.027834 11938 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:52:48.009383 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37822 (local address 127.0.104.3:46069)
1027 14:52:48.009563 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1027 14:52:48.009569 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:52:48.009590 (+ 21us) server_negotiation.cc:408] Connection header received
1027 14:52:48.009709 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:52:48.009716 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:52:48.009824 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:52:48.009940 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:52:48.011255 (+ 1315us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:48.012680 (+ 1425us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:52:48.013857 (+ 1177us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:52:48.014366 (+ 509us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:52:48.021790 (+ 7424us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:52:48.021831 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:52:48.021850 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:52:48.021893 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:52:48.025063 (+ 3170us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:48.025866 (+ 803us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:48.025874 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:48.025880 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:48.025978 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:52:48.026333 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:52:48.026337 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:52:48.026339 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:52:48.026676 (+ 337us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:52:48.026908 (+ 232us) server_negotiation.cc:1036] Waiting for connection context
1027 14:52:48.027265 (+ 357us) server_negotiation.cc:300] Negotiation successful
1027 14:52:48.027476 (+ 211us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
I20251027 14:52:48.028911 11919 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:48.148285 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11796
I20251027 14:52:48.163707 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 11429
I20251027 14:52:48.164820 11491 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:52:48.270483 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 11429
2025-10-27T14:52:48Z chronyd exiting
[ OK ] SecurityITest.TestRequireAuthenticationSecureCluster (5339 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegration
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:52:48 dist-test-slave-hxxt krb5kdc[11976](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:52:48 dist-test-slave-hxxt krb5kdc[11976](info): set up 2 sockets
Oct 27 14:52:48 dist-test-slave-hxxt krb5kdc[11976](info): commencing operation
krb5kdc: starting...
W20251027 14:52:50.145108 416 mini_kdc.cc:121] Time spent starting KDC: real 1.836s user 0.000s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:52:50 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576770, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:52:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:52:50Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/rangeradmin_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/rangeradmin_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/rangerlookup_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/rangerlookup_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
I20251027 14:52:50.467810 416 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuw8ijJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/postgres -l logfile start
2025-10-27 14:52:52.813 UTC [12007] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-27 14:52:52.813 UTC [12007] LOG: listening on IPv4 address "127.0.104.20", port 38717
2025-10-27 14:52:52.818 UTC [12007] LOG: listening on Unix socket "/tmp/.s.PGSQL.38717"
2025-10-27 14:52:52.822 UTC [12012] LOG: database system was shut down at 2025-10-27 14:52:51 UTC
2025-10-27 14:52:52.826 UTC [12007] LOG: database system is ready to accept connections
I20251027 14:52:54.588677 416 mini_postgres.cc:96] Postgres bound to 38717
2025-10-27 14:52:54.605 UTC [12018] FATAL: database "slave" does not exist
127.0.104.20:38717 - accepting connections
I20251027 14:52:54.606364 416 mini_ranger.cc:161] Starting Ranger...
I20251027 14:52:54.634842 416 mini_ranger.cc:84] Created miniranger Postgres user
I20251027 14:52:54.707548 416 mini_ranger.cc:87] Created ranger Postgres database
I20251027 14:52:54.707698 416 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-admin
2025-10-27 14:52:54,995 [I] DB FLAVOR :POSTGRES
2025-10-27 14:52:54,995 [I] --------- Verifying Ranger DB connection ---------
2025-10-27 14:52:54,996 [I] Checking connection..
2025-10-27 14:52:54,996 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-27 14:52:55,315 [I] Checking connection passed.
2025-10-27 14:52:55,315 [I] --------- Verifying version history table ---------
2025-10-27 14:52:55,315 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-27 14:52:55,633 [I] Table x_db_version_h does not exist in database ranger
2025-10-27 14:52:55,633 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-27 14:52:55,956 [I] Table x_db_version_h does not exist in database ranger
2025-10-27 14:52:55,956 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-27 14:52:55,956 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-27 14:52:56.266 UTC [12085] WARNING: there is no transaction in progress
2025-10-27 14:52:56,272 [I] create_dbversion_catalog.sql file imported successfully
2025-10-27 14:52:56,272 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-27 14:52:56,599 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-27 14:52:56,599 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-27 14:52:56,599 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-27 14:52:56,907 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-27 14:52:57,212 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hxxt.c.gcp-upstream.internal','N') ;"
2025-10-27 14:52:57,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-27 14:52:57,828 [I] Table x_portal_user does not exist in database ranger
2025-10-27 14:52:57,828 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-27 14:52:58,158 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-27 14:52:58,158 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-27 14:52:58,472 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-27 14:52:58,777 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-27 14:52:58,777 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-27 14:52:59.603 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.618 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.633 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.646 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.660 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.671 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.714 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.719 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.728 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.737 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.746 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.755 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.763 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.767 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.770 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:52:59.773 UTC [12202] WARNING: there is no transaction in progress
2025-10-27 14:53:00,135 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-27 14:53:00,135 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-27 14:53:00,443 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-27 14:53:00,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-27 14:53:00,753 [I] Table x_portal_user already exists in database 'ranger'
2025-10-27 14:53:00,753 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-27 14:53:01,050 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-27 14:53:01,051 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-27 14:53:01,360 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-27 14:53:01,675 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hxxt.c.gcp-upstream.internal';"
2025-10-27 14:53:01,992 [I] CORE_DB_SCHEMA import status has been updated
2025-10-27 14:53:01,993 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-27 14:53:02,322 [I] DB_PATCHES have already been applied
I20251027 14:53:02.328533 416 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuw8ijJ/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251027 14:53:02.328646 416 mini_ranger.cc:193] Using host: 127.0.104.20
I20251027 14:53:02.343009 416 mini_ranger.cc:227] Ranger admin URL: http://127.0.104.20:60601
Oct 27, 2025 2:53:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 27, 2025 2:53:02 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 27, 2025 2:53:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp
Oct 27, 2025 2:53:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 27, 2025 2:53:02 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 27, 2025 2:53:02 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 27, 2025 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp].
Oct 27, 2025 2:53:03 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 27, 2025 2:53:03 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-60601"]
Oct 27, 2025 2:53:03 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 27, 2025 2:53:03 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 27, 2025 2:53:03 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 27, 2025 2:53:03 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 27, 2025 2:53:03 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 27, 2025 2:53:04 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
I20251027 14:53:04.307920 416 mini_ranger.cc:160] Time spent starting Ranger: real 9.702s user 0.001s sys 0.040s
Oct 27, 2025 2:53:04 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 27, 2025 2:53:04 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 27, 2025 2:53:04 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
Oct 27, 2025 2:53:23 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 27, 2025 2:53:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIsv2
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.RoleREST
Oct 27, 2025 2:53:23 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 27, 2025 2:53:23 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 27, 2025 2:53:24 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 27, 2025 2:53:24 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-60601"]
I20251027 14:53:24.833477 416 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/rangerkms_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/rangerkms_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.104.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.104.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 27 14:53:24 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576804, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251027 14:53:25.007743 416 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251027 14:53:25.028528 416 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251027 14:53:25.135960 416 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251027 14:53:25.136075 416 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms
I20251027 14:53:25.136206 416 mini_ranger_kms.cc:227] Using postgres at 127.0.104.20:38717
2025-10-27 14:53:25,529 [I] DB FLAVOR :POSTGRES
2025-10-27 14:53:25,529 [I] --------- Verifying Ranger DB connection ---------
2025-10-27 14:53:25,529 [I] Checking connection
2025-10-27 14:53:25,529 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-27 14:53:25,858 [I] connection success
2025-10-27 14:53:25,858 [I] --------- Verifying Ranger DB tables ---------
2025-10-27 14:53:25,858 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-27 14:53:25,858 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-27 14:53:26,153 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-27 14:53:26,153 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-27 14:53:26,153 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-27 14:53:26,153 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:38717/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-27 14:53:26,461 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251027 14:53:26.948410 416 mini_ranger_kms.cc:310] Created kms service
I20251027 14:53:27.140094 416 mini_ranger_kms.cc:326] Created kudu user
I20251027 14:53:27.186808 416 mini_ranger_kms.cc:343] Created rangerkms user
Oct 27 14:53:27 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576804, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:53:27.549252 416 mini_ranger_kms.cc:384] Added ranger policy
I20251027 14:53:27.549551 416 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251027 14:53:27.549597 416 mini_ranger_kms.cc:240] Using host: 127.0.104.20
I20251027 14:53:27.571053 416 mini_ranger_kms.cc:276] Ranger KMS PID: 12394
I20251027 14:53:27.571214 416 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.104.20:44321
Oct 27, 2025 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 27, 2025 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 27, 2025 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 27, 2025 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 27, 2025 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-kms/ews/webapp].
Oct 27, 2025 2:53:28 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 27, 2025 2:53:28 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-44321"]
Oct 27, 2025 2:53:28 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 27, 2025 2:53:28 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 27, 2025 2:53:29 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 27, 2025 2:53:29 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
I20251027 14:53:29.599360 416 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.592s user 0.109s sys 0.296s
I20251027 14:53:29.599575 416 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251027 14:53:29.599771 416 mini_ranger_kms.cc:401] 127.0.104.20:44321/kms/v1/keys
Oct 27 14:53:29 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761576809, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.104.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Oct 27 14:53:30 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576809, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.104.20@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
Oct 27 14:53:30 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576809, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.104.20@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
Oct 27 14:53:32 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:53:32.767084 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46651
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:46651
--ranger_config_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:44321/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:53:33.100679 12447 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:53:33.101004 12447 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:53:33.101130 12447 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:53:33.109526 12447 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:53:33.109644 12447 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:53:33.109732 12447 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:53:33.109795 12447 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:53:33.109877 12447 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:53:33.120940 12447 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.104.20:44321/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:46651
--ranger_config_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:46651
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:53:33.122493 12447 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:53:33.123818 12447 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:53:33.131927 12453 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:33.131969 12452 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:33.132565 12455 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:53:33.133067 12447 server_base.cc:1047] running on GCE node
I20251027 14:53:33.133759 12447 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:53:33.135000 12447 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:53:33.136227 12447 hybrid_clock.cc:648] HybridClock initialized: now 1761576813136167 us; error 36 us; skew 500 ppm
I20251027 14:53:33.136575 12447 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:53:33 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:53:33.139931 12447 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:53:33.141440 12447 webserver.cc:492] Webserver started at http://127.0.104.62:38769/ using document root <none> and password file <none>
I20251027 14:53:33.142117 12447 fs_manager.cc:362] Metadata directory not provided
I20251027 14:53:33.142203 12447 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:53:33.142537 12447 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:53:33 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:53:33.190364 12447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "8c51d686147944d28bd66951221d6c57"
format_stamp: "Formatted at 2025-10-27 14:53:33 on dist-test-slave-hxxt"
server_key: "ab606cc2ee58a03d6ecdd9f84f54e1fc"
server_key_iv: "c495ffddfcd56ecea21d08b333a120a6"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:33.191236 12447 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "8c51d686147944d28bd66951221d6c57"
format_stamp: "Formatted at 2025-10-27 14:53:33 on dist-test-slave-hxxt"
server_key: "ab606cc2ee58a03d6ecdd9f84f54e1fc"
server_key_iv: "c495ffddfcd56ecea21d08b333a120a6"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:33.197036 12447 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:53:33.216890 12462 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:33.218353 12447 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251027 14:53:33.218514 12447 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "8c51d686147944d28bd66951221d6c57"
format_stamp: "Formatted at 2025-10-27 14:53:33 on dist-test-slave-hxxt"
server_key: "ab606cc2ee58a03d6ecdd9f84f54e1fc"
server_key_iv: "c495ffddfcd56ecea21d08b333a120a6"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:33.218735 12447 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:53:33.235436 12447 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:53:33.242944 12447 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:53:33.243211 12447 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:53:33.262733 12447 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:46651
I20251027 14:53:33.262753 12513 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:46651 every 8 connection(s)
I20251027 14:53:33.264433 12447 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:53:33.268918 12514 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:53:33.271320 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 12447
I20251027 14:53:33.271522 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:53:33.272053 416 external_mini_cluster.cc:1468] Setting key 814a46e8c4728a1744e7f3d2657ecbd6
I20251027 14:53:33.280320 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57: Bootstrap starting.
I20251027 14:53:33.284209 12514 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57: Neither blocks nor log segments found. Creating new log.
Oct 27 14:53:33 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:53:33.285292 12514 log.cc:826] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57: Log is configured to *not* fsync() on all Append() calls
I20251027 14:53:33.288915 12514 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57: No bootstrap required, opened a new log
I20251027 14:53:33.292598 12517 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:33.273930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50106 (local address 127.0.104.62:46651)
1027 14:53:33.275097 (+ 1167us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:33.275108 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:33.275147 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:53:33.276232 (+ 1085us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:33.276254 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:33.276680 (+ 426us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:33.277075 (+ 395us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:33.278732 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:33.280051 (+ 1319us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:33.281829 (+ 1778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:33.282294 (+ 465us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:33.285770 (+ 3476us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:33.286170 (+ 400us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:33.286197 (+ 27us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:33.286245 (+ 48us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:33.289113 (+ 2868us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:33.289703 (+ 590us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:33.289710 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:33.289717 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:33.289804 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:33.290088 (+ 284us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:33.290094 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:33.290098 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:33.290548 (+ 450us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:33.290838 (+ 290us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:33.291260 (+ 422us) server_negotiation.cc:300] Negotiation successful
1027 14:53:33.291605 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1003,"thread_start_us":842,"threads_started":1}
I20251027 14:53:33.294088 12514 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c51d686147944d28bd66951221d6c57" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46651 } }
I20251027 14:53:33.294405 12514 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:53:33.294498 12514 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c51d686147944d28bd66951221d6c57, State: Initialized, Role: FOLLOWER
I20251027 14:53:33.295029 12514 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c51d686147944d28bd66951221d6c57" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46651 } }
I20251027 14:53:33.295346 12514 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:53:33.295459 12514 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:53:33.295598 12514 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:53:33.298301 12514 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c51d686147944d28bd66951221d6c57" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46651 } }
I20251027 14:53:33.298775 12514 leader_election.cc:304] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8c51d686147944d28bd66951221d6c57; no voters:
I20251027 14:53:33.299299 12514 leader_election.cc:290] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:53:33.299407 12519 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:53:33.299959 12519 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [term 1 LEADER]: Becoming Leader. State: Replica: 8c51d686147944d28bd66951221d6c57, State: Running, Role: LEADER
I20251027 14:53:33.300431 12519 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c51d686147944d28bd66951221d6c57" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46651 } }
I20251027 14:53:33.301116 12514 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:53:33.303488 12521 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c51d686147944d28bd66951221d6c57. Latest consensus state: current_term: 1 leader_uuid: "8c51d686147944d28bd66951221d6c57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c51d686147944d28bd66951221d6c57" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46651 } } }
I20251027 14:53:33.303840 12521 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [sys.catalog]: This master's current role is: LEADER
I20251027 14:53:33.303838 12520 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c51d686147944d28bd66951221d6c57" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8c51d686147944d28bd66951221d6c57" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 46651 } } }
I20251027 14:53:33.304512 12520 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57 [sys.catalog]: This master's current role is: LEADER
I20251027 14:53:33.306319 12528 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:53:33.310590 12528 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:53:33.311301 12514 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251027 14:53:33.327421 12528 catalog_manager.cc:1357] Generated new cluster ID: 0eaaa3fbd76a454e976080d448c66c44
I20251027 14:53:33.327658 12528 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:53:33.388589 12528 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:53:33.390417 12528 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:53:33.404109 12528 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57: Generated new TSK 0
I20251027 14:53:33.405287 12528 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:53:34.164139 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:44321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46651
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:53:34 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761576814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20251027 14:53:34.569264 12560 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:53:34.569671 12560 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:53:34.569813 12560 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:53:34.580588 12560 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:53:34.580691 12560 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:53:34.580880 12560 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:53:34.596459 12560 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.104.20:44321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:46651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:53:34.598799 12560 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:53:34.600708 12560 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:53:34.615478 12570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:34.616972 12571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:34.625896 12573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:53:34.627007 12560 server_base.cc:1047] running on GCE node
I20251027 14:53:34.627738 12560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:53:34.629726 12560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:53:34.631006 12560 hybrid_clock.cc:648] HybridClock initialized: now 1761576814630923 us; error 51 us; skew 500 ppm
I20251027 14:53:34.631305 12560 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:53:34 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:53:34.634378 12560 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:53:34.639621 12560 webserver.cc:492] Webserver started at http://127.0.104.1:36147/ using document root <none> and password file <none>
I20251027 14:53:34.640359 12560 fs_manager.cc:362] Metadata directory not provided
I20251027 14:53:34.640502 12560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:53:34.640877 12560 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:53:34 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:53:34.669018 12560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "c66e80a6eeb34bb5a5c108fb78767294"
format_stamp: "Formatted at 2025-10-27 14:53:34 on dist-test-slave-hxxt"
server_key: "9e71c15446210ac5a5a8be92ec79ae70"
server_key_iv: "14a08f1f5eec071b9aad6fd083ef982b"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:34.672920 12560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "c66e80a6eeb34bb5a5c108fb78767294"
format_stamp: "Formatted at 2025-10-27 14:53:34 on dist-test-slave-hxxt"
server_key: "9e71c15446210ac5a5a8be92ec79ae70"
server_key_iv: "14a08f1f5eec071b9aad6fd083ef982b"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:34.680719 12560 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20251027 14:53:34.706239 12580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:34.708227 12560 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.002s sys 0.002s
I20251027 14:53:34.708451 12560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "c66e80a6eeb34bb5a5c108fb78767294"
format_stamp: "Formatted at 2025-10-27 14:53:34 on dist-test-slave-hxxt"
server_key: "9e71c15446210ac5a5a8be92ec79ae70"
server_key_iv: "14a08f1f5eec071b9aad6fd083ef982b"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:34.708750 12560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
Oct 27 14:53:34 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:53:34.737886 12560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:53:34.742321 12560 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:53:34.742591 12560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:53:34.743557 12560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:53:34.746188 12560 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:53:34.746351 12560 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251027 14:53:34.746543 12560 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:53:34.746694 12560 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:34.789635 12560 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:33943
I20251027 14:53:34.789654 12692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:33943 every 8 connection(s)
I20251027 14:53:34.791363 12560 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:53:34.793995 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 12560
I20251027 14:53:34.794216 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:53:34.794909 416 external_mini_cluster.cc:1468] Setting key b45beb7e6c0b20ef8f8294b8c653845a
Oct 27 14:53:34 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:53:34.829851 12695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:34.794305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:35597 (local address 127.0.104.62:46651)
1027 14:53:34.794784 (+ 479us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:34.794790 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:34.810013 (+ 15223us) server_negotiation.cc:408] Connection header received
1027 14:53:34.812776 (+ 2763us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:34.812784 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:34.812874 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:34.812979 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:34.815683 (+ 2704us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:34.817152 (+ 1469us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:34.818594 (+ 1442us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:34.819013 (+ 419us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:34.822755 (+ 3742us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:34.822805 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:34.822810 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:34.822849 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:34.826540 (+ 3691us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:34.827528 (+ 988us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:34.827535 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:34.827539 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:34.827636 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:34.828085 (+ 449us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:34.828089 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:34.828091 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:34.828321 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:34.828508 (+ 187us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:34.829251 (+ 743us) server_negotiation.cc:300] Negotiation successful
1027 14:53:34.829500 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":228,"threads_started":1}
I20251027 14:53:34.830883 12696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:34.794943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46651 (local address 127.0.104.1:35597)
1027 14:53:34.809800 (+ 14857us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:34.809863 (+ 63us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:34.812345 (+ 2482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:34.813731 (+ 1386us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:34.813809 (+ 78us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:34.814456 (+ 647us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:53:34.815447 (+ 991us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:34.815477 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:34.817368 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:34.817373 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:34.818433 (+ 1060us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:34.818447 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:34.819342 (+ 895us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:34.820146 (+ 804us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:53:34.820180 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:53:34.822540 (+ 2360us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:53:34.826747 (+ 4207us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:53:34.826758 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:53:34.826779 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:53:34.827365 (+ 586us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:53:34.827803 (+ 438us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:53:34.827807 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:53:34.827811 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:53:34.827968 (+ 157us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:53:34.828482 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:53:34.828499 (+ 17us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:53:34.828845 (+ 346us) client_negotiation.cc:769] Sending connection context
1027 14:53:34.829141 (+ 296us) client_negotiation.cc:240] Negotiation successful
1027 14:53:34.829615 (+ 474us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13279,"thread_start_us":13046,"threads_started":1}
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:53:34.832754 12693 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46651
I20251027 14:53:34.833132 12693 heartbeater.cc:461] Registering TS with master...
I20251027 14:53:34.834123 12693 heartbeater.cc:507] Master 127.0.104.62:46651 requested a full tablet report, sending...
I20251027 14:53:34.836776 12479 ts_manager.cc:194] Registered new tserver with Master: c66e80a6eeb34bb5a5c108fb78767294 (127.0.104.1:33943)
I20251027 14:53:34.838785 12479 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:35597
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:53:34.923718 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:44321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46651
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:53:34 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
W20251027 14:53:35.359867 12701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:53:35.360168 12701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:53:35.360301 12701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:53:35.368265 12701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:53:35.368369 12701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:53:35.368588 12701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:53:35.379086 12701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.104.20:44321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:46651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:53:35.380657 12701 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:53:35.381932 12701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:53:35.391016 12716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:35.391254 12715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:35.392118 12718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:53:35.392578 12701 server_base.cc:1047] running on GCE node
I20251027 14:53:35.393167 12701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:53:35.394058 12701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:53:35.395278 12701 hybrid_clock.cc:648] HybridClock initialized: now 1761576815395215 us; error 39 us; skew 500 ppm
I20251027 14:53:35.395555 12701 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:53:35 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:53:35.398522 12701 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:53:35.399915 12701 webserver.cc:492] Webserver started at http://127.0.104.2:34093/ using document root <none> and password file <none>
I20251027 14:53:35.400658 12701 fs_manager.cc:362] Metadata directory not provided
I20251027 14:53:35.400744 12701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:53:35.401083 12701 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:53:35 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:53:35.416317 12701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "6ce7d4072c924c75966f8c9972059eb6"
format_stamp: "Formatted at 2025-10-27 14:53:35 on dist-test-slave-hxxt"
server_key: "256000f0258a584fca29dde10841833c"
server_key_iv: "a03df16cdad0442628f59b9a9da79794"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:35.417143 12701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "6ce7d4072c924c75966f8c9972059eb6"
format_stamp: "Formatted at 2025-10-27 14:53:35 on dist-test-slave-hxxt"
server_key: "256000f0258a584fca29dde10841833c"
server_key_iv: "a03df16cdad0442628f59b9a9da79794"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:35.423094 12701 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.005s
I20251027 14:53:35.436800 12725 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:35.438180 12701 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:53:35.438355 12701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "6ce7d4072c924c75966f8c9972059eb6"
format_stamp: "Formatted at 2025-10-27 14:53:35 on dist-test-slave-hxxt"
server_key: "256000f0258a584fca29dde10841833c"
server_key_iv: "a03df16cdad0442628f59b9a9da79794"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:35.438562 12701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:53:35.458209 12701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:53:35.461683 12701 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:53:35.461931 12701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:53:35.462858 12701 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:53:35.464341 12701 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:53:35.464465 12701 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251027 14:53:35.464570 12701 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:53:35.464639 12701 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:35.496939 12701 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:34583
I20251027 14:53:35.496989 12837 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:34583 every 8 connection(s)
I20251027 14:53:35.498282 12701 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:53:35.499594 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 12701
I20251027 14:53:35.499778 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:53:35.500208 416 external_mini_cluster.cc:1468] Setting key 0f4a2ada0fa07265e003f7cb226ba916
Oct 27 14:53:35 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:53:35.517789 12840 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:35.501457 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:40977 (local address 127.0.104.62:46651)
1027 14:53:35.501958 (+ 501us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:35.501965 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:35.503010 (+ 1045us) server_negotiation.cc:408] Connection header received
1027 14:53:35.504543 (+ 1533us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:35.504550 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:35.504621 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:35.504733 (+ 112us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:35.506234 (+ 1501us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:35.507482 (+ 1248us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:35.508563 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:35.509014 (+ 451us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:35.511957 (+ 2943us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:35.511994 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:35.511997 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:35.512027 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:35.514929 (+ 2902us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:35.515751 (+ 822us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:35.515756 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:35.515758 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:35.515834 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:35.516171 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:35.516174 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:35.516176 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:35.516410 (+ 234us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:35.516524 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:35.517343 (+ 819us) server_negotiation.cc:300] Negotiation successful
1027 14:53:35.517541 (+ 198us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":343,"thread_start_us":219,"threads_started":1}
I20251027 14:53:35.519270 12841 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:35.502066 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46651 (local address 127.0.104.2:40977)
1027 14:53:35.502835 (+ 769us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:35.502880 (+ 45us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:35.504232 (+ 1352us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:35.504931 (+ 699us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:35.504939 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:35.505381 (+ 442us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:53:35.506061 (+ 680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:35.506079 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:35.507614 (+ 1535us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:35.507617 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:35.508422 (+ 805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:35.508435 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:35.508708 (+ 273us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:35.509410 (+ 702us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:53:35.509442 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:53:35.511733 (+ 2291us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:53:35.515071 (+ 3338us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:53:35.515081 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:53:35.515101 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:53:35.515651 (+ 550us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:53:35.515926 (+ 275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:53:35.515930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:53:35.515934 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:53:35.516090 (+ 156us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:53:35.516523 (+ 433us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:53:35.516531 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:53:35.516919 (+ 388us) client_negotiation.cc:769] Sending connection context
1027 14:53:35.517293 (+ 374us) client_negotiation.cc:240] Negotiation successful
1027 14:53:35.517760 (+ 467us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":528,"thread_start_us":365,"threads_started":1}
I20251027 14:53:35.520763 12838 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46651
I20251027 14:53:35.521155 12838 heartbeater.cc:461] Registering TS with master...
I20251027 14:53:35.522158 12838 heartbeater.cc:507] Master 127.0.104.62:46651 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
I20251027 14:53:35.523834 12479 ts_manager.cc:194] Registered new tserver with Master: 6ce7d4072c924c75966f8c9972059eb6 (127.0.104.2:34583)
I20251027 14:53:35.524766 12479 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:40977
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:53:35.588657 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:44321/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:46651
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:35139
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:53:35.842854 12693 heartbeater.cc:499] Master 127.0.104.62:46651 was elected leader, sending a full tablet report...
W20251027 14:53:35.917605 12846 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:53:35.917878 12846 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:53:35.917975 12846 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:53:35.925940 12846 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:53:35.926039 12846 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:53:35.926659 12846 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:53:35.936861 12846 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35139
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.104.20:44321/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:46651
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:53:35.938331 12846 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:53:35.939613 12846 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:53:35.949251 12851 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:35.949275 12852 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:53:35.949853 12854 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:53:35.950327 12846 server_base.cc:1047] running on GCE node
I20251027 14:53:35.950894 12846 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:53:35.951714 12846 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:53:35.952934 12846 hybrid_clock.cc:648] HybridClock initialized: now 1761576815952872 us; error 40 us; skew 500 ppm
I20251027 14:53:35.953200 12846 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:53:35 dist-test-slave-hxxt krb5kdc[11976](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:53:35.956151 12846 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:53:35.957679 12846 webserver.cc:492] Webserver started at http://127.0.104.3:46199/ using document root <none> and password file <none>
I20251027 14:53:35.958376 12846 fs_manager.cc:362] Metadata directory not provided
I20251027 14:53:35.958461 12846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:53:35.958796 12846 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:53:35 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:53:35.973801 12846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2"
format_stamp: "Formatted at 2025-10-27 14:53:35 on dist-test-slave-hxxt"
server_key: "42ea512a864d57d2102e73130cf32616"
server_key_iv: "52695076251657b4676af57efd32d369"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:35.974599 12846 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2"
format_stamp: "Formatted at 2025-10-27 14:53:35 on dist-test-slave-hxxt"
server_key: "42ea512a864d57d2102e73130cf32616"
server_key_iv: "52695076251657b4676af57efd32d369"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:35.980011 12846 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:53:35.994938 12861 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:35.996533 12846 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:53:35.996697 12846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2"
format_stamp: "Formatted at 2025-10-27 14:53:35 on dist-test-slave-hxxt"
server_key: "42ea512a864d57d2102e73130cf32616"
server_key_iv: "52695076251657b4676af57efd32d369"
server_key_version: "kuduclusterkey@0"
I20251027 14:53:35.996924 12846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:53:36.035218 12846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:53:36.039186 12846 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:53:36.039456 12846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:53:36.040437 12846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:53:36.041956 12846 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:53:36.042048 12846 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:36.042145 12846 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:53:36.042210 12846 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:53:36.074853 12846 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34253
I20251027 14:53:36.074899 12973 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34253 every 8 connection(s)
I20251027 14:53:36.076141 12846 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:53:36.078867 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 12846
I20251027 14:53:36.079046 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:53:36.079519 416 external_mini_cluster.cc:1468] Setting key 68c07b00ac677df83a04593926d90c3c
Oct 27 14:53:36 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:53:36.093465 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.078696 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:51377 (local address 127.0.104.62:46651)
1027 14:53:36.079145 (+ 449us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.079152 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.080400 (+ 1248us) server_negotiation.cc:408] Connection header received
1027 14:53:36.082180 (+ 1780us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.082186 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.082270 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.082357 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:36.083857 (+ 1500us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.084773 (+ 916us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.085758 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.086041 (+ 283us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.088475 (+ 2434us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:36.088501 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:36.088505 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:36.088533 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:36.090960 (+ 2427us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.091519 (+ 559us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.091523 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.091525 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.091587 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.091848 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.091851 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.091853 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.092064 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:36.092170 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.092947 (+ 777us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.093128 (+ 181us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":211,"threads_started":1}
I20251027 14:53:36.094494 12977 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.079252 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:46651 (local address 127.0.104.3:51377)
1027 14:53:36.080043 (+ 791us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:36.080089 (+ 46us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:36.081937 (+ 1848us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:36.082589 (+ 652us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:36.082598 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:36.083046 (+ 448us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:53:36.083716 (+ 670us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.083735 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.084889 (+ 1154us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.084893 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:36.085643 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.085653 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.085874 (+ 221us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.086461 (+ 587us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:53:36.086488 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:53:36.088291 (+ 1803us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:53:36.091061 (+ 2770us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:53:36.091067 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:53:36.091080 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:53:36.091426 (+ 346us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:53:36.091660 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:53:36.091663 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:53:36.091666 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:53:36.091779 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:53:36.092151 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:53:36.092157 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:53:36.092476 (+ 319us) client_negotiation.cc:769] Sending connection context
1027 14:53:36.092859 (+ 383us) client_negotiation.cc:240] Negotiation successful
1027 14:53:36.093243 (+ 384us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":332,"threads_started":1}
I20251027 14:53:36.095625 12974 heartbeater.cc:344] Connected to a master server at 127.0.104.62:46651
I20251027 14:53:36.095938 12974 heartbeater.cc:461] Registering TS with master...
I20251027 14:53:36.096699 12974 heartbeater.cc:507] Master 127.0.104.62:46651 requested a full tablet report, sending...
I20251027 14:53:36.098001 12479 ts_manager.cc:194] Registered new tserver with Master: a9b321dbf18c4a2eb8dec3d6a49578b2 (127.0.104.3:34253)
I20251027 14:53:36.098774 12479 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:51377
I20251027 14:53:36.106549 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:53:36.122189 12976 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.111213 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50116 (local address 127.0.104.62:46651)
1027 14:53:36.111348 (+ 135us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.111353 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.111774 (+ 421us) server_negotiation.cc:408] Connection header received
1027 14:53:36.111904 (+ 130us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.111908 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.111965 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.112040 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:36.113640 (+ 1600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.114502 (+ 862us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.115779 (+ 1277us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.116186 (+ 407us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.117863 (+ 1677us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:36.117893 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:36.117896 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:36.117922 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:36.120317 (+ 2395us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.120935 (+ 618us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.120938 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.120940 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.121002 (+ 62us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.121217 (+ 215us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.121220 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.121221 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.121410 (+ 189us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:36.121605 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.121801 (+ 196us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.121978 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:53:36.125572 12479 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50116:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:53:36.131021 12479 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:53:36.154004 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.143247 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60674 (local address 127.0.104.1:33943)
1027 14:53:36.144078 (+ 831us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.144088 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.144802 (+ 714us) server_negotiation.cc:408] Connection header received
1027 14:53:36.145199 (+ 397us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.145208 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.145507 (+ 299us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.145698 (+ 191us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:53:36.147191 (+ 1493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.149182 (+ 1991us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.151564 (+ 2382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.152779 (+ 1215us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.152948 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.153488 (+ 540us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.153697 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":697,"thread_start_us":594,"threads_started":1}
I20251027 14:53:36.154769 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.143208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:33943 (local address 127.0.0.1:60674)
1027 14:53:36.144614 (+ 1406us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:36.144645 (+ 31us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:36.144958 (+ 313us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:36.146131 (+ 1173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:36.146139 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:36.146175 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:53:36.146986 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.147008 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.149472 (+ 2464us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.149478 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:36.151367 (+ 1889us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.151383 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.153090 (+ 1707us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.153156 (+ 66us) client_negotiation.cc:769] Sending connection context
1027 14:53:36.154214 (+ 1058us) client_negotiation.cc:240] Negotiation successful
1027 14:53:36.154430 (+ 216us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1180,"thread_start_us":1041,"threads_started":1}
I20251027 14:53:36.160413 12628 tablet_service.cc:1505] Processing CreateTablet for tablet f3f11ef1119f42e98f7ccd7e97798361 (DEFAULT_TABLE table=test-table [id=d489795af7aa4c66aebb3c0aea8aa4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:53:36.161265 12988 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.144780 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34253 (local address 127.0.0.1:33504)
1027 14:53:36.155125 (+ 10345us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:36.155145 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:36.155258 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:36.156337 (+ 1079us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:36.156342 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:36.156365 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:53:36.156847 (+ 482us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.156861 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.158974 (+ 2113us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.158979 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:36.160659 (+ 1680us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.160673 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.160815 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.160839 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:53:36.160897 (+ 58us) client_negotiation.cc:240] Negotiation successful
1027 14:53:36.161011 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10270,"thread_start_us":12268,"threads_started":1}
I20251027 14:53:36.162151 12628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f11ef1119f42e98f7ccd7e97798361. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:53:36.162156 12990 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.145115 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:34583 (local address 127.0.0.1:57098)
1027 14:53:36.155464 (+ 10349us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:36.155480 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:36.155570 (+ 90us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:36.157327 (+ 1757us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:36.157332 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:36.157358 (+ 26us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:53:36.157916 (+ 558us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.157926 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.160060 (+ 2134us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.160064 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:36.161502 (+ 1438us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.161516 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.161688 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.161712 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:53:36.161772 (+ 60us) client_negotiation.cc:240] Negotiation successful
1027 14:53:36.161879 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":10263,"thread_start_us":10132,"threads_started":1}
I20251027 14:53:36.162598 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.145054 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33504 (local address 127.0.104.3:34253)
1027 14:53:36.154941 (+ 9887us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.154950 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.155392 (+ 442us) server_negotiation.cc:408] Connection header received
1027 14:53:36.155511 (+ 119us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.155519 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.155792 (+ 273us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.156037 (+ 245us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:53:36.157038 (+ 1001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.158800 (+ 1762us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.160824 (+ 2024us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.161910 (+ 1086us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.162040 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.162158 (+ 118us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.162417 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9728,"thread_start_us":9873,"threads_started":1}
I20251027 14:53:36.163478 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.145611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57098 (local address 127.0.104.2:34583)
1027 14:53:36.156001 (+ 10390us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.156009 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.156037 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:53:36.156128 (+ 91us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.156136 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.156369 (+ 233us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.157355 (+ 986us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:53:36.158045 (+ 690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.159877 (+ 1832us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.161827 (+ 1950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.162836 (+ 1009us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.162964 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.163073 (+ 109us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.163226 (+ 153us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10229,"thread_start_us":11058,"threads_started":1}
I20251027 14:53:36.166846 12773 tablet_service.cc:1505] Processing CreateTablet for tablet f3f11ef1119f42e98f7ccd7e97798361 (DEFAULT_TABLE table=test-table [id=d489795af7aa4c66aebb3c0aea8aa4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:53:36.167030 12909 tablet_service.cc:1505] Processing CreateTablet for tablet f3f11ef1119f42e98f7ccd7e97798361 (DEFAULT_TABLE table=test-table [id=d489795af7aa4c66aebb3c0aea8aa4f2]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:53:36.168021 12773 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f11ef1119f42e98f7ccd7e97798361. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:53:36.168519 12909 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f3f11ef1119f42e98f7ccd7e97798361. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:53:36.178933 12993 tablet_bootstrap.cc:492] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Bootstrap starting.
I20251027 14:53:36.180794 12994 tablet_bootstrap.cc:492] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Bootstrap starting.
I20251027 14:53:36.182835 12993 tablet_bootstrap.cc:654] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Neither blocks nor log segments found. Creating new log.
I20251027 14:53:36.183892 12993 log.cc:826] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Log is configured to *not* fsync() on all Append() calls
I20251027 14:53:36.184515 12994 tablet_bootstrap.cc:654] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Neither blocks nor log segments found. Creating new log.
I20251027 14:53:36.185604 12995 tablet_bootstrap.cc:492] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Bootstrap starting.
I20251027 14:53:36.186301 12994 log.cc:826] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Log is configured to *not* fsync() on all Append() calls
I20251027 14:53:36.188288 12993 tablet_bootstrap.cc:492] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: No bootstrap required, opened a new log
I20251027 14:53:36.188566 12993 ts_tablet_manager.cc:1403] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20251027 14:53:36.189021 12994 tablet_bootstrap.cc:492] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: No bootstrap required, opened a new log
I20251027 14:53:36.189250 12994 ts_tablet_manager.cc:1403] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Time spent bootstrapping tablet: real 0.009s user 0.001s sys 0.006s
I20251027 14:53:36.193830 12993 raft_consensus.cc:359] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.194145 12993 raft_consensus.cc:385] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:53:36.193990 12994 raft_consensus.cc:359] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.194269 12993 raft_consensus.cc:740] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c66e80a6eeb34bb5a5c108fb78767294, State: Initialized, Role: FOLLOWER
I20251027 14:53:36.194342 12994 raft_consensus.cc:385] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:53:36.194453 12994 raft_consensus.cc:740] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6ce7d4072c924c75966f8c9972059eb6, State: Initialized, Role: FOLLOWER
I20251027 14:53:36.194928 12993 consensus_queue.cc:260] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.195135 12994 consensus_queue.cc:260] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.196578 12993 ts_tablet_manager.cc:1434] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Time spent starting tablet: real 0.008s user 0.007s sys 0.001s
I20251027 14:53:36.198014 12995 tablet_bootstrap.cc:654] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Neither blocks nor log segments found. Creating new log.
I20251027 14:53:36.198318 12838 heartbeater.cc:499] Master 127.0.104.62:46651 was elected leader, sending a full tablet report...
I20251027 14:53:36.198913 12994 ts_tablet_manager.cc:1434] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
I20251027 14:53:36.200335 12995 log.cc:826] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Log is configured to *not* fsync() on all Append() calls
I20251027 14:53:36.203773 12995 tablet_bootstrap.cc:492] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: No bootstrap required, opened a new log
I20251027 14:53:36.204025 12995 ts_tablet_manager.cc:1403] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Time spent bootstrapping tablet: real 0.019s user 0.007s sys 0.000s
I20251027 14:53:36.209535 12995 raft_consensus.cc:359] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.209975 12995 raft_consensus.cc:385] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:53:36.210088 12995 raft_consensus.cc:740] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a9b321dbf18c4a2eb8dec3d6a49578b2, State: Initialized, Role: FOLLOWER
I20251027 14:53:36.210752 12995 consensus_queue.cc:260] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.211674 12974 heartbeater.cc:499] Master 127.0.104.62:46651 was elected leader, sending a full tablet report...
I20251027 14:53:36.212224 12995 ts_tablet_manager.cc:1434] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Time spent starting tablet: real 0.008s user 0.009s sys 0.000s
W20251027 14:53:36.250584 12839 tablet.cc:2378] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:53:36.293913 12694 tablet.cc:2378] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:53:36.326303 12998 raft_consensus.cc:493] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:53:36.326552 12998 raft_consensus.cc:515] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
W20251027 14:53:36.327621 12975 tablet.cc:2378] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:53:36.328239 12998 leader_election.cc:290] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6ce7d4072c924c75966f8c9972059eb6 (127.0.104.2:34583), a9b321dbf18c4a2eb8dec3d6a49578b2 (127.0.104.3:34253)
I20251027 14:53:36.335222 13002 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.328704 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:34583 (local address 127.0.104.1:42029)
1027 14:53:36.329904 (+ 1200us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:36.329919 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:36.330074 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:36.330410 (+ 336us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:36.330415 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:36.330445 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:53:36.331131 (+ 686us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.331141 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.332601 (+ 1460us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.332637 (+ 36us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:36.334469 (+ 1832us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.334484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.334689 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.334707 (+ 18us) client_negotiation.cc:769] Sending connection context
1027 14:53:36.334763 (+ 56us) client_negotiation.cc:240] Negotiation successful
1027 14:53:36.334885 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1097,"thread_start_us":884,"threads_started":1}
I20251027 14:53:36.335747 13003 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.328769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34253 (local address 127.0.104.1:50117)
1027 14:53:36.330026 (+ 1257us) negotiation.cc:107] Waiting for socket to connect
1027 14:53:36.330039 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1027 14:53:36.330154 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:53:36.330488 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:53:36.330493 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:53:36.330530 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:53:36.331177 (+ 647us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.331187 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.332691 (+ 1504us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.332696 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:53:36.334469 (+ 1773us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:53:36.334484 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.334662 (+ 178us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.334693 (+ 31us) client_negotiation.cc:769] Sending connection context
1027 14:53:36.334763 (+ 70us) client_negotiation.cc:240] Negotiation successful
1027 14:53:36.334885 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1169,"thread_start_us":985,"threads_started":1}
I20251027 14:53:36.335762 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.329297 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:42029 (local address 127.0.104.2:34583)
1027 14:53:36.329489 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.329497 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.330019 (+ 522us) server_negotiation.cc:408] Connection header received
1027 14:53:36.330230 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.330237 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.330303 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.330371 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:53:36.331271 (+ 900us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.332350 (+ 1079us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.334616 (+ 2266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.335332 (+ 716us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.335372 (+ 40us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.335432 (+ 60us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.335517 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:53:36.336980 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3f11ef1119f42e98f7ccd7e97798361" candidate_uuid: "c66e80a6eeb34bb5a5c108fb78767294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce7d4072c924c75966f8c9972059eb6" is_pre_election: true
I20251027 14:53:36.337432 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.328608 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:50117 (local address 127.0.104.3:34253)
1027 14:53:36.328790 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.328796 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.330123 (+ 1327us) server_negotiation.cc:408] Connection header received
1027 14:53:36.330266 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.330270 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.330336 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.330422 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:53:36.331326 (+ 904us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.332507 (+ 1181us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.336032 (+ 3525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.337048 (+ 1016us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.337098 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.337167 (+ 69us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.337255 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251027 14:53:36.337443 12793 raft_consensus.cc:2468] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c66e80a6eeb34bb5a5c108fb78767294 in term 0.
I20251027 14:53:36.338060 12584 leader_election.cc:304] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 6ce7d4072c924c75966f8c9972059eb6, c66e80a6eeb34bb5a5c108fb78767294; no voters:
I20251027 14:53:36.338238 12929 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3f11ef1119f42e98f7ccd7e97798361" candidate_uuid: "c66e80a6eeb34bb5a5c108fb78767294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" is_pre_election: true
I20251027 14:53:36.338400 12998 raft_consensus.cc:2804] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:53:36.338555 12998 raft_consensus.cc:493] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:53:36.338650 12998 raft_consensus.cc:3060] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:53:36.338814 12929 raft_consensus.cc:2468] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c66e80a6eeb34bb5a5c108fb78767294 in term 0.
I20251027 14:53:36.341140 12998 raft_consensus.cc:515] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.341802 12998 leader_election.cc:290] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [CANDIDATE]: Term 1 election: Requested vote from peers 6ce7d4072c924c75966f8c9972059eb6 (127.0.104.2:34583), a9b321dbf18c4a2eb8dec3d6a49578b2 (127.0.104.3:34253)
I20251027 14:53:36.342213 12793 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3f11ef1119f42e98f7ccd7e97798361" candidate_uuid: "c66e80a6eeb34bb5a5c108fb78767294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6ce7d4072c924c75966f8c9972059eb6"
I20251027 14:53:36.342308 12929 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f3f11ef1119f42e98f7ccd7e97798361" candidate_uuid: "c66e80a6eeb34bb5a5c108fb78767294" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2"
I20251027 14:53:36.342417 12793 raft_consensus.cc:3060] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:53:36.342505 12929 raft_consensus.cc:3060] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:53:36.345885 12929 raft_consensus.cc:2468] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c66e80a6eeb34bb5a5c108fb78767294 in term 1.
I20251027 14:53:36.345885 12793 raft_consensus.cc:2468] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c66e80a6eeb34bb5a5c108fb78767294 in term 1.
I20251027 14:53:36.346357 12583 leader_election.cc:304] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a9b321dbf18c4a2eb8dec3d6a49578b2, c66e80a6eeb34bb5a5c108fb78767294; no voters:
I20251027 14:53:36.346695 12998 raft_consensus.cc:2804] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:53:36.347091 12998 raft_consensus.cc:697] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 1 LEADER]: Becoming Leader. State: Replica: c66e80a6eeb34bb5a5c108fb78767294, State: Running, Role: LEADER
I20251027 14:53:36.347529 12998 consensus_queue.cc:237] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } }
I20251027 14:53:36.352795 12479 catalog_manager.cc:5649] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 reported cstate change: term changed from 0 to 1, leader changed from <none> to c66e80a6eeb34bb5a5c108fb78767294 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "c66e80a6eeb34bb5a5c108fb78767294" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "c66e80a6eeb34bb5a5c108fb78767294" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 33943 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 } health_report { overall_health: UNKNOWN } } }
I20251027 14:53:36.394377 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.389590 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60678 (local address 127.0.104.1:33943)
1027 14:53:36.389769 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.389775 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.389798 (+ 23us) server_negotiation.cc:408] Connection header received
1027 14:53:36.389895 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.389899 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.389972 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.390100 (+ 128us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:53:36.390729 (+ 629us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.391630 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.392888 (+ 1258us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.393205 (+ 317us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.393363 (+ 158us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:53:36.393901 (+ 538us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:53:36.394015 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.394097 (+ 82us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.394180 (+ 83us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:53:36.403262 12793 raft_consensus.cc:1275] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 1 FOLLOWER]: Refusing update from remote peer c66e80a6eeb34bb5a5c108fb78767294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:53:36.403286 12929 raft_consensus.cc:1275] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 1 FOLLOWER]: Refusing update from remote peer c66e80a6eeb34bb5a5c108fb78767294: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:53:36.404325 12998 consensus_queue.cc:1048] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a9b321dbf18c4a2eb8dec3d6a49578b2" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:53:36.404656 13004 consensus_queue.cc:1048] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6ce7d4072c924c75966f8c9972059eb6" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 34583 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:53:36.420877 13008 mvcc.cc:204] Tried to move back new op lower bound from 7215418639976071168 to 7215418639765860352. Current Snapshot: MvccSnapshot[applied={T|T < 7215418639976071168}]
I20251027 14:53:36.444062 12478 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50116:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:53:36.444284 12478 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50116:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:53:36.448204 12478 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 8c51d686147944d28bd66951221d6c57: Sending DeleteTablet for 3 replicas of tablet f3f11ef1119f42e98f7ccd7e97798361
I20251027 14:53:36.449477 12628 tablet_service.cc:1552] Processing DeleteTablet for tablet f3f11ef1119f42e98f7ccd7e97798361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:53:36 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:60674
I20251027 14:53:36.449463 12909 tablet_service.cc:1552] Processing DeleteTablet for tablet f3f11ef1119f42e98f7ccd7e97798361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:53:36 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:33504
I20251027 14:53:36.450183 12773 tablet_service.cc:1552] Processing DeleteTablet for tablet f3f11ef1119f42e98f7ccd7e97798361 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:53:36 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:57098
I20251027 14:53:36.450469 13018 tablet_replica.cc:333] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: stopping tablet replica
I20251027 14:53:36.451066 13019 tablet_replica.cc:333] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: stopping tablet replica
I20251027 14:53:36.451355 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 12560
I20251027 14:53:36.452153 13019 raft_consensus.cc:2243] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:53:36.452732 13019 raft_consensus.cc:2272] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:53:36.455111 13019 ts_tablet_manager.cc:1916] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:53:36.458910 13018 raft_consensus.cc:2243] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:53:36.459434 13018 raft_consensus.cc:2272] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:53:36.462668 13018 ts_tablet_manager.cc:1916] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:53:36.462962 13019 ts_tablet_manager.cc:1929] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:53:36.463156 13019 log.cc:1199] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/f3f11ef1119f42e98f7ccd7e97798361
I20251027 14:53:36.463697 13019 ts_tablet_manager.cc:1950] T f3f11ef1119f42e98f7ccd7e97798361 P 6ce7d4072c924c75966f8c9972059eb6: Deleting consensus metadata
I20251027 14:53:36.465608 12466 catalog_manager.cc:4985] TS 6ce7d4072c924c75966f8c9972059eb6 (127.0.104.2:34583): tablet f3f11ef1119f42e98f7ccd7e97798361 (table test-table [id=d489795af7aa4c66aebb3c0aea8aa4f2]) successfully deleted
I20251027 14:53:36.466977 13017 tablet_replica.cc:333] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: stopping tablet replica
I20251027 14:53:36.467453 13017 raft_consensus.cc:2243] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:53:36.468225 13017 raft_consensus.cc:2272] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:53:36.469470 13018 ts_tablet_manager.cc:1929] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:53:36.469616 13018 log.cc:1199] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/f3f11ef1119f42e98f7ccd7e97798361
I20251027 14:53:36.470044 13018 ts_tablet_manager.cc:1950] T f3f11ef1119f42e98f7ccd7e97798361 P a9b321dbf18c4a2eb8dec3d6a49578b2: Deleting consensus metadata
I20251027 14:53:36.470566 13017 ts_tablet_manager.cc:1916] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:53:36.471750 12465 catalog_manager.cc:4985] TS a9b321dbf18c4a2eb8dec3d6a49578b2 (127.0.104.3:34253): tablet f3f11ef1119f42e98f7ccd7e97798361 (table test-table [id=d489795af7aa4c66aebb3c0aea8aa4f2]) successfully deleted
Oct 27 14:53:36 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:53:36.478389 13017 ts_tablet_manager.cc:1929] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:53:36.478577 13017 log.cc:1199] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegration.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/f3f11ef1119f42e98f7ccd7e97798361
I20251027 14:53:36.479113 13017 ts_tablet_manager.cc:1950] T f3f11ef1119f42e98f7ccd7e97798361 P c66e80a6eeb34bb5a5c108fb78767294: Deleting consensus metadata
I20251027 14:53:36.481027 12466 catalog_manager.cc:4985] TS c66e80a6eeb34bb5a5c108fb78767294 (127.0.104.1:33943): tablet f3f11ef1119f42e98f7ccd7e97798361 (table test-table [id=d489795af7aa4c66aebb3c0aea8aa4f2]) successfully deleted
I20251027 14:53:36.482421 12987 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.452680 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60680 (local address 127.0.104.1:33943)
1027 14:53:36.452834 (+ 154us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.452840 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.466373 (+ 13533us) server_negotiation.cc:408] Connection header received
1027 14:53:36.466615 (+ 242us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.466622 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.466714 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.466815 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:36.469239 (+ 2424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.470752 (+ 1513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.472463 (+ 1711us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.472981 (+ 518us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.476142 (+ 3161us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:36.476201 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:36.476216 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:36.476255 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:36.479470 (+ 3215us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.480103 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.480110 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.480113 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.480192 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.480636 (+ 444us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.480642 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.480645 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.481020 (+ 375us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:36.481341 (+ 321us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.481868 (+ 527us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.482114 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:53:36.483122 12688 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:53:36.595192 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 12560
I20251027 14:53:36.610954 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 12701
Oct 27 14:53:36 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:53:36.624987 12992 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.611699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:57104 (local address 127.0.104.2:34583)
1027 14:53:36.611863 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.611868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.611887 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:53:36.612082 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.612088 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.612157 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.612239 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:36.613631 (+ 1392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.614518 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.615679 (+ 1161us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.616074 (+ 395us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.619646 (+ 3572us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:36.619692 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:36.619708 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:36.619741 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:36.622474 (+ 2733us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.623115 (+ 641us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.623123 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.623125 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.623201 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.623516 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.623522 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.623524 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.623810 (+ 286us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:36.624113 (+ 303us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.624405 (+ 292us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.624648 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:53:36.625686 12833 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:53:36.713901 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 12701
I20251027 14:53:36.727389 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 12846
Oct 27 14:53:36 dist-test-slave-hxxt krb5kdc[11976](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576812, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:53:36.741223 12991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:53:36.728069 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33506 (local address 127.0.104.3:34253)
1027 14:53:36.728226 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1027 14:53:36.728231 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:53:36.728248 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:53:36.728488 (+ 240us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:53:36.728493 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:53:36.728565 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:53:36.728645 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:53:36.730025 (+ 1380us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.730870 (+ 845us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:53:36.731922 (+ 1052us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:53:36.732261 (+ 339us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:53:36.735817 (+ 3556us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:53:36.735854 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:53:36.735871 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:53:36.735907 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:53:36.738931 (+ 3024us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.739564 (+ 633us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.739571 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.739574 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.739643 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:53:36.739851 (+ 208us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:53:36.739854 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:53:36.739856 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:53:36.740097 (+ 241us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:53:36.740334 (+ 237us) server_negotiation.cc:1036] Waiting for connection context
1027 14:53:36.740737 (+ 403us) server_negotiation.cc:300] Negotiation successful
1027 14:53:36.740914 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:53:36.741837 12969 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:53:36.828807 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 12846
I20251027 14:53:36.841869 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 12447
I20251027 14:53:36.842965 12509 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:53:36.952463 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 12447
I20251027 14:53:36.973064 416 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251027 14:53:37.308189 416 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251027 14:53:37.308434 416 mini_ranger.cc:66] Stopping Ranger...
I20251027 14:53:37.643287 416 mini_ranger.cc:68] Stopped Ranger
2025-10-27 14:53:37.643 UTC [12007] LOG: received smart shutdown request
2025-10-27 14:53:37.646 UTC [12007] LOG: background worker "logical replication launcher" (PID 12015) exited with exit code 1
2025-10-27 14:53:37.646 UTC [12010] LOG: shutting down
2025-10-27 14:53:37.648 UTC [12010] LOG: checkpoint starting: shutdown immediate
2025-10-27 14:53:38.366 UTC [12010] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.032 s, sync=0.682 s, total=0.720 s; sync files=1108, longest=0.004 s, average=0.001 s; distance=12074 kB, estimate=12074 kB; lsn=0/2045148, redo lsn=0/2045148
2025-10-27 14:53:38.373 UTC [12007] LOG: database system is shut down
2025-10-27T14:53:38Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegration (50210 ms)
[ RUN ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:53:38 dist-test-slave-hxxt krb5kdc[13033](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:53:38 dist-test-slave-hxxt krb5kdc[13033](info): set up 2 sockets
Oct 27 14:53:38 dist-test-slave-hxxt krb5kdc[13033](info): commencing operation
krb5kdc: starting...
W20251027 14:53:40.360728 416 mini_kdc.cc:121] Time spent starting KDC: real 1.841s user 0.001s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:53:40 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576820, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:53:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:53:40Z Disabled control of system clock
WARNING: no policy specified for rangeradmin/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangeradmin/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangeradmin/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/rangeradmin_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal rangeradmin/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/rangeradmin_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for rangerlookup/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerlookup/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerlookup/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/rangerlookup_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal rangerlookup/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/rangerlookup_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
I20251027 14:53:40.664783 416 mini_postgres.cc:62] Running initdb...
The files belonging to this database system will be owned by user "slave".
This user must also own the server process.
The database cluster will be initialized with locale "C".
The default database encoding has accordingly been set to "SQL_ASCII".
The default text search configuration will be set to "english".
Data page checksums are disabled.
creating directory /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/postgres ... ok
creating subdirectories ... ok
selecting dynamic shared memory implementation ... posix
selecting default "max_connections" ... 100
selecting default "shared_buffers" ... 128MB
selecting default time zone ... Etc/UTC
creating configuration files ... ok
running bootstrap script ... ok
performing post-bootstrap initialization ... ok
syncing data to disk ... ok
initdb: warning: enabling "trust" authentication for local connections
initdb: hint: You can change this by editing pg_hba.conf or using the option -A, or --auth-local and --auth-host, the next time you run initdb.
Success. You can now start the database server using:
/tmp/dist-test-taskuw8ijJ/build/asan/bin/postgres/pg_ctl -D /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/postgres -l logfile start
2025-10-27 14:53:42.909 UTC [13064] LOG: starting PostgreSQL 17.2 on x86_64-pc-linux-gnu, compiled by gcc (Ubuntu 7.5.0-3ubuntu1~18.04) 7.5.0, 64-bit
2025-10-27 14:53:42.909 UTC [13064] LOG: listening on IPv4 address "127.0.104.20", port 50715
2025-10-27 14:53:42.913 UTC [13064] LOG: listening on Unix socket "/tmp/.s.PGSQL.50715"
2025-10-27 14:53:42.917 UTC [13069] LOG: database system was shut down at 2025-10-27 14:53:41 UTC
2025-10-27 14:53:42.921 UTC [13064] LOG: database system is ready to accept connections
I20251027 14:53:44.680433 416 mini_postgres.cc:96] Postgres bound to 50715
2025-10-27 14:53:44.696 UTC [13075] FATAL: database "slave" does not exist
127.0.104.20:50715 - accepting connections
I20251027 14:53:44.697077 416 mini_ranger.cc:161] Starting Ranger...
I20251027 14:53:44.724685 416 mini_ranger.cc:84] Created miniranger Postgres user
I20251027 14:53:44.797995 416 mini_ranger.cc:87] Created ranger Postgres database
I20251027 14:53:44.798107 416 mini_ranger.cc:178] Starting Ranger out of /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-admin
2025-10-27 14:53:45,083 [I] DB FLAVOR :POSTGRES
2025-10-27 14:53:45,084 [I] --------- Verifying Ranger DB connection ---------
2025-10-27 14:53:45,084 [I] Checking connection..
2025-10-27 14:53:45,084 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select 1;"
2025-10-27 14:53:45,383 [I] Checking connection passed.
2025-10-27 14:53:45,383 [I] --------- Verifying version history table ---------
2025-10-27 14:53:45,383 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-27 14:53:45,692 [I] Table x_db_version_h does not exist in database ranger
2025-10-27 14:53:45,692 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-27 14:53:45,998 [I] Table x_db_version_h does not exist in database ranger
2025-10-27 14:53:45,998 [I] Importing x_db_version_h table schema to database ranger from file: create_dbversion_catalog.sql
2025-10-27 14:53:45,998 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/create_dbversion_catalog.sql
2025-10-27 14:53:46.317 UTC [13142] WARNING: there is no transaction in progress
2025-10-27 14:53:46,323 [I] create_dbversion_catalog.sql file imported successfully
2025-10-27 14:53:46,324 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_db_version_h') as temp;"
2025-10-27 14:53:46,614 [I] Table x_db_version_h already exists in database 'ranger'
2025-10-27 14:53:46,614 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-27 14:53:46,615 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'Y';"
2025-10-27 14:53:46,945 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'CORE_DB_SCHEMA' and active = 'N';"
2025-10-27 14:53:47,273 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "insert into x_db_version_h (version, inst_at, inst_by, updated_at, updated_by,active) values ('CORE_DB_SCHEMA', current_timestamp, 'Ranger 2.1.0', current_timestamp, 'dist-test-slave-hxxt.c.gcp-upstream.internal','N') ;"
2025-10-27 14:53:47,573 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-27 14:53:47,914 [I] Table x_portal_user does not exist in database ranger
2025-10-27 14:53:47,914 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-27 14:53:48,222 [I] Table x_policy_ref_group does not exist in database ranger
2025-10-27 14:53:48,223 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-27 14:53:48,526 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-27 14:53:48,853 [I] Importing DB schema to database ranger from file: ranger_core_db_postgres.sql
2025-10-27 14:53:48,854 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/db/postgres/optimized/current/ranger_core_db_postgres.sql
2025-10-27 14:53:49.639 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.649 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.660 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.670 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.681 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.691 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.741 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.747 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.759 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.771 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.783 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.792 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.800 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.804 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.807 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:49.811 UTC [13259] WARNING: there is no transaction in progress
2025-10-27 14:53:50,196 [I] ranger_core_db_postgres.sql file imported successfully
2025-10-27 14:53:50,196 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set inst_by='Ranger 2.1.0' where active='Y' and updated_by='localhost';"
2025-10-27 14:53:50,515 [I] Patches status entries updated from base ranger version to current installed ranger version:Ranger 2.1.0
2025-10-27 14:53:50,515 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_portal_user') as temp;"
2025-10-27 14:53:50,840 [I] Table x_portal_user already exists in database 'ranger'
2025-10-27 14:53:50,840 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='ranger' and table_name = 'x_policy_ref_group') as temp;"
2025-10-27 14:53:51,134 [I] Table x_policy_ref_group already exists in database 'ranger'
2025-10-27 14:53:51,134 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and active = 'Y';"
2025-10-27 14:53:51,444 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'JAVA_PATCHES' and active = 'Y';"
2025-10-27 14:53:51,769 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "update x_db_version_h set active='Y' where version='CORE_DB_SCHEMA' and active='N' and updated_by='dist-test-slave-hxxt.c.gcp-upstream.internal';"
2025-10-27 14:53:52,087 [I] CORE_DB_SCHEMA import status has been updated
2025-10-27 14:53:52,087 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/ranger -u miniranger -p '********' -noheader -trim -c \; -query "select version from x_db_version_h where version = 'DB_PATCHES' and inst_by = 'Ranger 2.1.0' and active = 'Y';"
2025-10-27 14:53:52,425 [I] DB_PATCHES have already been applied
I20251027 14:53:52.430792 416 mini_ranger.cc:191] Using Ranger class path: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-admin:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*:/tmp/dist-test-taskuw8ijJ/thirdparty/src/hadoop-3.4.1/*:/tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp
I20251027 14:53:52.430887 416 mini_ranger.cc:193] Using host: 127.0.104.20
I20251027 14:53:52.444329 416 mini_ranger.cc:227] Ranger admin URL: http://127.0.104.20:58407
Oct 27, 2025 2:53:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 27, 2025 2:53:52 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 27, 2025 2:53:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Deriving webapp folder from catalina.base property. folder=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp
Oct 27, 2025 2:53:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp, webAppName = /
Oct 27, 2025 2:53:52 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp] .....
Oct 27, 2025 2:53:52 PM org.apache.catalina.core.StandardContext setPath
WARNING: A context path must either be an empty string or start with a '/' and do not end with a '/'. The path [/] does not meet these criteria and has been changed to []
Oct 27, 2025 2:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-admin/ews/webapp].
Oct 27, 2025 2:53:53 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : miniranger
Oct 27, 2025 2:53:53 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-58407"]
Oct 27, 2025 2:53:53 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 27, 2025 2:53:53 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 27, 2025 2:53:53 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/javax.servlet-api-3.1.0.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/servlet/Servlet.class
Oct 27, 2025 2:53:53 PM org.apache.catalina.loader.WebappClassLoaderBase validateJarFile
INFO: validateJarFile(/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/WEB-INF/lib/tomcat-el-api-7.0.105.jar) - jar not loaded. See Servlet Spec 3.0, section 10.7.2. Offending class: javax/el/Expression.class
Oct 27, 2025 2:53:53 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 27, 2025 2:53:54 PM org.apache.catalina.startup.TaglibUriRule body
INFO: TLD skipped. URI: urn:com:sun:jersey:api:view is already defined
Oct 27, 2025 2:53:54 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 27, 2025 2:53:54 PM org.apache.catalina.core.ApplicationContext log
INFO: Set web app root system property: 'webapp.root' = [/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-admin/ews/webapp/]
Oct 27, 2025 2:53:54 PM org.apache.catalina.core.ApplicationContext log
INFO: Initializing Spring root WebApplicationContext
I20251027 14:53:54.435618 416 mini_ranger.cc:160] Time spent starting Ranger: real 9.739s user 0.000s sys 0.042s
Oct 27, 2025 2:54:12 PM com.sun.jersey.api.core.PackagesResourceConfig init
INFO: Scanning for root resource and provider classes in the packages:
org.apache.ranger.rest
org.apache.ranger.common
xa.rest
Oct 27, 2025 2:54:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Root resource classes found:
class org.apache.ranger.rest.TagREST
class org.apache.ranger.rest.MetricsREST
class org.apache.ranger.rest.RoleREST
class org.apache.ranger.rest.XUserREST
class org.apache.ranger.rest.PublicAPIs
class org.apache.ranger.rest.ServiceREST
class org.apache.ranger.rest.XAuditREST
class org.apache.ranger.rest.XKeyREST
class org.apache.ranger.rest.UserREST
class org.apache.ranger.rest.AssetREST
class org.apache.ranger.rest.SecurityZoneREST
class org.apache.ranger.rest.PublicAPIsv2
Oct 27, 2025 2:54:12 PM com.sun.jersey.api.core.ScanningResourceConfig logClasses
INFO: Provider classes found:
class org.apache.ranger.common.RangerJAXBContextResolver
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.servlet.SpringServlet getContext
INFO: Using default applicationContext
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, assetREST, of type org.apache.ranger.rest.AssetREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, metricsREST, of type org.apache.ranger.rest.MetricsREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIs, of type org.apache.ranger.rest.PublicAPIs as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, publicAPIsv2, of type org.apache.ranger.rest.PublicAPIsv2 as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, roleREST, of type org.apache.ranger.rest.RoleREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, securityZoneREST, of type org.apache.ranger.rest.SecurityZoneREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, serviceREST, of type org.apache.ranger.rest.ServiceREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, tagREST, of type org.apache.ranger.rest.TagREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, userREST, of type org.apache.ranger.rest.UserREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XAuditREST, of type org.apache.ranger.rest.XAuditREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XKeyREST, of type org.apache.ranger.rest.XKeyREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.spi.spring.container.SpringComponentProviderFactory registerSpringBeans
INFO: Registering Spring bean, XUserREST, of type org.apache.ranger.rest.XUserREST as a root resource class
Oct 27, 2025 2:54:12 PM com.sun.jersey.server.impl.application.WebApplicationImpl _initiate
INFO: Initiating Jersey application, version 'Jersey: 1.19.3 10/24/2016 03:58 PM'
Oct 27, 2025 2:54:13 PM com.sun.jersey.spi.inject.Errors processErrorMessages
WARNING: The following warnings have been detected with resource and/or provider classes:
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInExcel(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInJson(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse,java.lang.Boolean), MUST return a non-void type.
WARNING: A HTTP GET method, public void org.apache.ranger.rest.ServiceREST.getPoliciesInCsv(javax.servlet.http.HttpServletRequest,javax.servlet.http.HttpServletResponse) throws java.io.IOException, MUST return a non-void type.
Oct 27, 2025 2:54:13 PM org.apache.coyote.AbstractProtocol start
INFO: Starting ProtocolHandler ["http-bio-58407"]
I20251027 14:54:14.251051 416 mini_ranger.cc:261] Created Kudu service
WARNING: no policy specified for rangerkms/127.0.104.20@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "rangerkms/127.0.104.20@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal rangerkms/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/rangerkms_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal rangerkms/127.0.104.20@KRBTEST.COM with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/rangerkms_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for HTTP/127.0.104.20@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.104.20@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
Entry for principal HTTP/127.0.104.20@KRBTEST.COM with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/HTTP_127.0.104.20@KRBTEST.COM.keytab.
WARNING: no policy specified for keyadmin@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "keyadmin@KRBTEST.COM" created.
Oct 27 14:54:14 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576854, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for keyadmin@KRBTEST.COM:
I20251027 14:54:14.412351 416 mini_ranger_kms.cc:208] Starting Ranger KMS...
I20251027 14:54:14.432526 416 mini_ranger_kms.cc:78] Created minirangerkms Postgres user
I20251027 14:54:14.545526 416 mini_ranger_kms.cc:81] Created rangerkms Postgres database
I20251027 14:54:14.545634 416 mini_ranger_kms.cc:226] Starting Ranger KMS out of /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms
I20251027 14:54:14.545670 416 mini_ranger_kms.cc:227] Using postgres at 127.0.104.20:50715
2025-10-27 14:54:14,884 [I] DB FLAVOR :POSTGRES
2025-10-27 14:54:14,884 [I] --------- Verifying Ranger DB connection ---------
2025-10-27 14:54:14,885 [I] Checking connection
2025-10-27 14:54:14,885 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "SELECT 1;"
2025-10-27 14:54:15,209 [I] connection success
2025-10-27 14:54:15,210 [I] --------- Verifying Ranger DB tables ---------
2025-10-27 14:54:15,210 [I] Verifying table ranger_masterkey in database rangerkms
2025-10-27 14:54:15,210 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -query "select * from (select table_name from information_schema.tables where table_catalog='rangerkms' and table_name = 'ranger_masterkey') as temp;"
2025-10-27 14:54:15,540 [I] Table ranger_masterkey does not exist in database rangerkms
2025-10-27 14:54:15,540 [I] --------- Importing Ranger Core DB Schema ---------
2025-10-27 14:54:15,540 [I] Importing db schema to database rangerkms from file: kms_core_db_postgres.sql
2025-10-27 14:54:15,540 [JISQL] /usr/lib/jvm/java-1.8.0-openjdk-amd64/bin/java -cp /tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/jisql/lib/* org.apache.util.sql.Jisql -driver postgresql -cstring jdbc:postgresql://127.0.104.20:50715/rangerkms -u rangerkms -p '********' -noheader -trim -c \; -input /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/db/postgres/kms_core_db_postgres.sql
2025-10-27 14:54:15,870 [I] kms_core_db_postgres.sql DB schema imported successfully
I20251027 14:54:16.349182 416 mini_ranger_kms.cc:310] Created kms service
I20251027 14:54:16.558965 416 mini_ranger_kms.cc:326] Created kudu user
I20251027 14:54:16.606407 416 mini_ranger_kms.cc:343] Created rangerkms user
Oct 27 14:54:16 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576854, etypes {rep=17 tkt=17 ses=17}, keyadmin@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:16.991990 416 mini_ranger_kms.cc:384] Added ranger policy
I20251027 14:54:16.992281 416 mini_ranger_kms.cc:238] Using RangerKMS classpath: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms:/tmp/dist-test-taskuw8ijJ/build/asan/bin/postgresql.jar:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/WEB-INF/classes/lib/*:/tmp/dist-test-taskuw8ijJ/thirdparty/src/ranger-2.1.0-kms/ews/webapp/lib/*:/usr/lib/jvm/java-1.8.0-openjdk-amd64/lib/*
I20251027 14:54:16.992326 416 mini_ranger_kms.cc:240] Using host: 127.0.104.20
I20251027 14:54:17.006255 416 mini_ranger_kms.cc:276] Ranger KMS PID: 13451
I20251027 14:54:17.006388 416 mini_ranger_kms.cc:277] Ranger KMS URL: http://127.0.104.20:50689
Oct 27, 2025 2:54:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getKeyManagers
WARNING: Config 'ranger.keystore.file' or 'ranger.service.https.attrib.keystore.file' is not found or contains blank value
Oct 27, 2025 2:54:17 PM org.apache.ranger.server.tomcat.EmbeddedServer getTrustManagers
WARNING: Config 'ranger.truststore.file' is not found or contains blank value!
Oct 27, 2025 2:54:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Webapp file =/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/ews/webapp, webAppName = /kms
Oct 27, 2025 2:54:17 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Adding webapp [/kms] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/ews/webapp] .....
Oct 27, 2025 2:54:18 PM org.apache.ranger.server.tomcat.EmbeddedServer start
INFO: Finished init of webapp [/kms] = path [/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-kms/ews/webapp].
Oct 27, 2025 2:54:18 PM org.apache.ranger.server.tomcat.EmbeddedServer startServer
INFO: Server Name : minirangerkms
Oct 27, 2025 2:54:18 PM org.apache.coyote.AbstractProtocol init
INFO: Initializing ProtocolHandler ["http-bio-50689"]
Oct 27, 2025 2:54:18 PM org.apache.catalina.core.StandardService startInternal
INFO: Starting service [Tomcat]
Oct 27, 2025 2:54:18 PM org.apache.catalina.core.StandardEngine startInternal
INFO: Starting Servlet Engine: Apache Tomcat/7.0.105
Oct 27, 2025 2:54:18 PM org.apache.catalina.startup.ContextConfig getDefaultWebXmlFragment
INFO: No global web.xml found
Oct 27, 2025 2:54:18 PM org.apache.catalina.startup.TldConfig execute
INFO: At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.
Oct 27 14:54:18 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761576858, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.104.20@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:19.018978 416 mini_ranger_kms.cc:207] Time spent starting Ranger KMS: real 4.607s user 0.083s sys 0.265s
I20251027 14:54:19.019177 416 mini_ranger_kms.cc:397] {"name":"kuduclusterkey","cipher":"AES/CTR/NoPadding","length":128,"description":"kuduclusterkey"}
I20251027 14:54:19.019351 416 mini_ranger_kms.cc:401] 127.0.104.20:50689/kms/v1/keys
Oct 27 14:54:19 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576858, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.104.20@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
Oct 27 14:54:19 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576858, etypes {rep=17 tkt=17 ses=17}, rangerkms/127.0.104.20@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
Oct 27 14:54:21 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:22.084076 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35299
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35299
--ranger_config_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:22.421255 13504 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:22.421568 13504 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:22.421666 13504 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:22.429836 13504 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:22.429950 13504 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:22.430009 13504 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:22.430055 13504 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:22.430112 13504 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:22.441898 13504 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
--ranger_kms_url=127.0.104.20:50689/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35299
--ranger_config_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:22.443409 13504 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:22.444799 13504 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:22.453066 13509 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:22.453104 13512 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:22.453123 13510 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:22.453806 13504 server_base.cc:1047] running on GCE node
I20251027 14:54:22.454538 13504 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:22.455899 13504 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:22.457113 13504 hybrid_clock.cc:648] HybridClock initialized: now 1761576862457043 us; error 40 us; skew 500 ppm
I20251027 14:54:22.457412 13504 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:22 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:22.460568 13504 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:22.462073 13504 webserver.cc:492] Webserver started at http://127.0.104.62:35649/ using document root <none> and password file <none>
I20251027 14:54:22.462759 13504 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:22.462854 13504 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:22.463186 13504 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:54:22 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576862, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:22.525902 13504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11"
format_stamp: "Formatted at 2025-10-27 14:54:22 on dist-test-slave-hxxt"
server_key: "4f6b98dc1dceea29a91cdce486872394"
server_key_iv: "90434c41c55a3cb0fb39f3c3ad1d5a77"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:22.526741 13504 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11"
format_stamp: "Formatted at 2025-10-27 14:54:22 on dist-test-slave-hxxt"
server_key: "4f6b98dc1dceea29a91cdce486872394"
server_key_iv: "90434c41c55a3cb0fb39f3c3ad1d5a77"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:22.532308 13504 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:54:22.551540 13519 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:22.553025 13504 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:22.553194 13504 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11"
format_stamp: "Formatted at 2025-10-27 14:54:22 on dist-test-slave-hxxt"
server_key: "4f6b98dc1dceea29a91cdce486872394"
server_key_iv: "90434c41c55a3cb0fb39f3c3ad1d5a77"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:22.553426 13504 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:22.569914 13504 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:22.578213 13504 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:22.578498 13504 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:22.597975 13504 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35299
I20251027 14:54:22.598008 13570 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35299 every 8 connection(s)
I20251027 14:54:22.599411 13504 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:22.603719 13571 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:22.607511 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 13504
I20251027 14:54:22.607748 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:54:22.608474 416 external_mini_cluster.cc:1468] Setting key 6541b2f637e4c0038336f6ceacad09be
I20251027 14:54:22.615895 13571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Bootstrap starting.
I20251027 14:54:22.619756 13571 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:22.621161 13571 log.cc:826] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Log is configured to *not* fsync() on all Append() calls
Oct 27 14:54:22 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:22.624873 13571 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: No bootstrap required, opened a new log
I20251027 14:54:22.630239 13574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:22.610776 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50152 (local address 127.0.104.62:35299)
1027 14:54:22.611622 (+ 846us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:22.611634 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:22.611676 (+ 42us) server_negotiation.cc:408] Connection header received
1027 14:54:22.613238 (+ 1562us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:22.613262 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:22.613719 (+ 457us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:22.614074 (+ 355us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:22.615911 (+ 1837us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:22.617401 (+ 1490us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:22.619008 (+ 1607us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:22.619489 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:22.623457 (+ 3968us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:22.623585 (+ 128us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:22.623603 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:22.623637 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:22.626438 (+ 2801us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:22.627233 (+ 795us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:22.627240 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:22.627247 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:22.627338 (+ 91us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:22.627668 (+ 330us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:22.627672 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:22.627674 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:22.628128 (+ 454us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:22.628368 (+ 240us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:22.628833 (+ 465us) server_negotiation.cc:300] Negotiation successful
1027 14:54:22.629192 (+ 359us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":632,"thread_start_us":412,"threads_started":1}
I20251027 14:54:22.631054 13571 raft_consensus.cc:359] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:22.631469 13571 raft_consensus.cc:385] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:22.631595 13571 raft_consensus.cc:740] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5cb0e1b8784b0a93f0f708dcc74f11, State: Initialized, Role: FOLLOWER
I20251027 14:54:22.632197 13571 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:22.632427 13571 raft_consensus.cc:399] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:54:22.632568 13571 raft_consensus.cc:493] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:54:22.632723 13571 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:22.635092 13571 raft_consensus.cc:515] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:22.635618 13571 leader_election.cc:304] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5cb0e1b8784b0a93f0f708dcc74f11; no voters:
I20251027 14:54:22.636168 13571 leader_election.cc:290] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:54:22.636276 13576 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:22.636881 13576 raft_consensus.cc:697] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 LEADER]: Becoming Leader. State: Replica: eb5cb0e1b8784b0a93f0f708dcc74f11, State: Running, Role: LEADER
I20251027 14:54:22.637367 13576 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:22.638072 13571 sys_catalog.cc:565] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:22.639981 13577 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } } }
I20251027 14:54:22.639994 13578 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5cb0e1b8784b0a93f0f708dcc74f11. Latest consensus state: current_term: 1 leader_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } } }
I20251027 14:54:22.640399 13578 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:22.640273 13577 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:22.641423 13582 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:22.647645 13582 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:22.655520 13571 ranger_client.cc:318] Using new properties file: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251027 14:54:22.665215 13582 catalog_manager.cc:1357] Generated new cluster ID: 707a8a153fc94aacb4fe560396d48a69
I20251027 14:54:22.665488 13582 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:54:22.699560 13582 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:54:22.701511 13582 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:54:22.715762 13582 catalog_manager.cc:6022] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Generated new TSK 0
I20251027 14:54:22.716938 13582 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:23.505970 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35299
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:54:23 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
W20251027 14:54:23.894299 13617 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:23.894620 13617 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:23.894781 13617 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:23.903470 13617 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:23.903575 13617 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:23.903805 13617 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:54:23.916791 13617 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
--ranger_kms_url=127.0.104.20:50689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:23.918808 13617 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:23.920194 13617 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:23.929114 13627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:23.929809 13628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:23.941120 13630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:23.941666 13617 server_base.cc:1047] running on GCE node
I20251027 14:54:23.942409 13617 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:23.943665 13617 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:23.944943 13617 hybrid_clock.cc:648] HybridClock initialized: now 1761576863944869 us; error 47 us; skew 500 ppm
I20251027 14:54:23.945269 13617 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:23 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:23.948494 13617 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:54:23.950369 13617 webserver.cc:492] Webserver started at http://127.0.104.1:37719/ using document root <none> and password file <none>
I20251027 14:54:23.951113 13617 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:23.951192 13617 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:23.951532 13617 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:54:23 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:23.978058 13617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "aa4dbca8af074b39a37447f902c50f79"
format_stamp: "Formatted at 2025-10-27 14:54:23 on dist-test-slave-hxxt"
server_key: "8c36fe4a77ab21d9b614725c43320441"
server_key_iv: "1766de88e0fe3939e9a8ae80e6461412"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:23.979364 13617 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "aa4dbca8af074b39a37447f902c50f79"
format_stamp: "Formatted at 2025-10-27 14:54:23 on dist-test-slave-hxxt"
server_key: "8c36fe4a77ab21d9b614725c43320441"
server_key_iv: "1766de88e0fe3939e9a8ae80e6461412"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:24.001950 13617 fs_manager.cc:696] Time spent creating directory manager: real 0.022s user 0.008s sys 0.012s
I20251027 14:54:24.029078 13637 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:24.034452 13617 fs_manager.cc:730] Time spent opening block manager: real 0.015s user 0.003s sys 0.007s
I20251027 14:54:24.034677 13617 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "aa4dbca8af074b39a37447f902c50f79"
format_stamp: "Formatted at 2025-10-27 14:54:23 on dist-test-slave-hxxt"
server_key: "8c36fe4a77ab21d9b614725c43320441"
server_key_iv: "1766de88e0fe3939e9a8ae80e6461412"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:24.034971 13617 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:24.101863 13617 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:24.106981 13617 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:24.107350 13617 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:24.114764 13617 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
Oct 27 14:54:24 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:24.123270 13617 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:24.123399 13617 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:24.123520 13617 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:24.123595 13617 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:24.212728 13617 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39393
I20251027 14:54:24.212746 13749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39393 every 8 connection(s)
I20251027 14:54:24.214448 13617 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:54:24.217236 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 13617
I20251027 14:54:24.217512 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:54:24.218346 416 external_mini_cluster.cc:1468] Setting key a61cd4605d810bf39c3e587669182e6b
Oct 27 14:54:24 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:24.246470 13753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:24.219356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:57845 (local address 127.0.104.62:35299)
1027 14:54:24.226737 (+ 7381us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:24.226743 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:24.230151 (+ 3408us) server_negotiation.cc:408] Connection header received
1027 14:54:24.230230 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:24.230236 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:24.230325 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:24.230449 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:24.233720 (+ 3271us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.235066 (+ 1346us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:24.236578 (+ 1512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.236975 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:24.240668 (+ 3693us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:24.240713 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:24.240716 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:24.240747 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:24.243388 (+ 2641us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:24.244050 (+ 662us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:24.244054 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:24.244056 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:24.244133 (+ 77us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:24.244497 (+ 364us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:24.244503 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:24.244507 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:24.244807 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:24.244932 (+ 125us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:24.245865 (+ 933us) server_negotiation.cc:300] Negotiation successful
1027 14:54:24.246097 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":7260,"thread_start_us":7137,"threads_started":1}
I20251027 14:54:24.251196 13752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:24.218726 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35299 (local address 127.0.104.1:57845)
1027 14:54:24.227340 (+ 8614us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:24.227411 (+ 71us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:24.229536 (+ 2125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:24.230939 (+ 1403us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:24.230955 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:24.231719 (+ 764us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:24.232867 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:24.232897 (+ 30us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.235282 (+ 2385us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:24.235288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:24.236416 (+ 1128us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:24.236432 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.237333 (+ 901us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:24.238169 (+ 836us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:24.238202 (+ 33us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:24.240463 (+ 2261us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:24.243514 (+ 3051us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:24.243523 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:24.243542 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:24.243927 (+ 385us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:24.244229 (+ 302us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:24.244232 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:24.244235 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:24.244347 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:24.245017 (+ 670us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:24.245025 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:24.245412 (+ 387us) client_negotiation.cc:769] Sending connection context
1027 14:54:24.246921 (+ 1509us) client_negotiation.cc:240] Negotiation successful
1027 14:54:24.247399 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8281,"thread_start_us":18518,"threads_started":1}
I20251027 14:54:24.253273 13750 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35299
I20251027 14:54:24.253662 13750 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:24.254631 13750 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:24.256788 13536 ts_manager.cc:194] Registered new tserver with Master: aa4dbca8af074b39a37447f902c50f79 (127.0.104.1:39393)
I20251027 14:54:24.258342 13536 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:57845
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:24.353541 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35299
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:54:24 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576863, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
W20251027 14:54:24.790270 13758 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:24.790560 13758 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:24.790691 13758 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:24.798802 13758 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:24.798894 13758 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:24.799067 13758 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:54:24.809543 13758 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
--ranger_kms_url=127.0.104.20:50689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:24.811064 13758 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:24.812390 13758 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:24.821542 13773 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:24.822029 13772 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:24.822506 13775 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:24.823606 13758 server_base.cc:1047] running on GCE node
I20251027 14:54:24.824162 13758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:24.825099 13758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:24.826321 13758 hybrid_clock.cc:648] HybridClock initialized: now 1761576864826226 us; error 75 us; skew 500 ppm
I20251027 14:54:24.826599 13758 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:24 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:24.829639 13758 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:54:24.831305 13758 webserver.cc:492] Webserver started at http://127.0.104.2:34887/ using document root <none> and password file <none>
I20251027 14:54:24.832027 13758 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:24.832105 13758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:24.832614 13758 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:54:24 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:24.848573 13758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "e081745b922a44bca5e1093d6adc72ea"
format_stamp: "Formatted at 2025-10-27 14:54:24 on dist-test-slave-hxxt"
server_key: "a82d07deaef7eff175c6c998d29d553f"
server_key_iv: "50d4ac6ef19907be3a75db9d2f49169c"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:24.849606 13758 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "e081745b922a44bca5e1093d6adc72ea"
format_stamp: "Formatted at 2025-10-27 14:54:24 on dist-test-slave-hxxt"
server_key: "a82d07deaef7eff175c6c998d29d553f"
server_key_iv: "50d4ac6ef19907be3a75db9d2f49169c"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:24.855394 13758 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:54:24.874468 13782 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:24.875999 13758 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20251027 14:54:24.876176 13758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "e081745b922a44bca5e1093d6adc72ea"
format_stamp: "Formatted at 2025-10-27 14:54:24 on dist-test-slave-hxxt"
server_key: "a82d07deaef7eff175c6c998d29d553f"
server_key_iv: "50d4ac6ef19907be3a75db9d2f49169c"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:24.876441 13758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:24.905025 13758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:24.908819 13758 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:24.909072 13758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:24.910020 13758 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:24.911485 13758 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:24.911584 13758 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:24.911689 13758 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:24.911756 13758 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:24.946810 13758 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:32981
I20251027 14:54:24.946902 13894 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:32981 every 8 connection(s)
I20251027 14:54:24.948176 13758 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:54:24.953706 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 13758
I20251027 14:54:24.953910 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:54:24.954366 416 external_mini_cluster.cc:1468] Setting key 82072df484ddc5db5fece3b2f8b77f15
Oct 27 14:54:24 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576864, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:24.969358 13897 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:24.950977 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:47347 (local address 127.0.104.62:35299)
1027 14:54:24.951491 (+ 514us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:24.951498 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:24.952466 (+ 968us) server_negotiation.cc:408] Connection header received
1027 14:54:24.954421 (+ 1955us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:24.954430 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:24.954526 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:24.954664 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:24.956436 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.957988 (+ 1552us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:24.959458 (+ 1470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.960431 (+ 973us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:24.963431 (+ 3000us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:24.963494 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:24.963500 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:24.963543 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:24.966546 (+ 3003us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:24.967209 (+ 663us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:24.967215 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:24.967218 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:24.967294 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:24.967613 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:24.967618 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:24.967622 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:24.967845 (+ 223us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:24.967972 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:24.968732 (+ 760us) server_negotiation.cc:300] Negotiation successful
1027 14:54:24.968987 (+ 255us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":388,"thread_start_us":228,"threads_started":1}
I20251027 14:54:24.970408 13898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:24.951489 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35299 (local address 127.0.104.2:47347)
1027 14:54:24.952254 (+ 765us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:24.952312 (+ 58us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:24.954023 (+ 1711us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:24.954940 (+ 917us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:24.954951 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:24.955550 (+ 599us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:24.956232 (+ 682us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:24.956251 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.958164 (+ 1913us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:24.958169 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:24.959299 (+ 1130us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:24.959313 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:24.959585 (+ 272us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:24.960352 (+ 767us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:24.960405 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:24.963201 (+ 2796us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:24.966681 (+ 3480us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:24.966690 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:24.966709 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:24.967099 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:24.967391 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:24.967395 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:24.967398 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:24.967515 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:24.967961 (+ 446us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:24.967969 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:24.968282 (+ 313us) client_negotiation.cc:769] Sending connection context
1027 14:54:24.968628 (+ 346us) client_negotiation.cc:240] Negotiation successful
1027 14:54:24.969077 (+ 449us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":507,"thread_start_us":334,"threads_started":1}
I20251027 14:54:24.971844 13895 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35299
I20251027 14:54:24.972206 13895 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:24.973088 13895 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:24.974650 13536 ts_manager.cc:194] Registered new tserver with Master: e081745b922a44bca5e1093d6adc72ea (127.0.104.2:32981)
I20251027 14:54:24.975464 13536 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:47347
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:25.047287 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35299
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:25.261490 13750 heartbeater.cc:499] Master 127.0.104.62:35299 was elected leader, sending a full tablet report...
W20251027 14:54:25.364975 13903 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:25.365273 13903 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:25.365424 13903 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:25.373437 13903 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:25.373561 13903 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:25.373744 13903 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:54:25.383920 13903 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
--ranger_kms_url=127.0.104.20:50689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:25.385481 13903 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:25.386794 13903 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:25.395565 13908 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:25.395596 13909 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:25.395851 13911 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:25.396518 13903 server_base.cc:1047] running on GCE node
I20251027 14:54:25.397049 13903 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:25.397902 13903 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:25.399134 13903 hybrid_clock.cc:648] HybridClock initialized: now 1761576865399074 us; error 37 us; skew 500 ppm
I20251027 14:54:25.399410 13903 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:25 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:25.402420 13903 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:54:25.403950 13903 webserver.cc:492] Webserver started at http://127.0.104.3:39043/ using document root <none> and password file <none>
I20251027 14:54:25.404726 13903 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:25.404808 13903 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:25.405140 13903 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
Oct 27 14:54:25 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:25.420015 13903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "3b71fb20372346c6a13f6a341a6fff7f"
format_stamp: "Formatted at 2025-10-27 14:54:25 on dist-test-slave-hxxt"
server_key: "8bb0224daec19ab00d6e0a561b0d5526"
server_key_iv: "ce861c7514b59509af0604f60d11d278"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:25.420893 13903 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "3b71fb20372346c6a13f6a341a6fff7f"
format_stamp: "Formatted at 2025-10-27 14:54:25 on dist-test-slave-hxxt"
server_key: "8bb0224daec19ab00d6e0a561b0d5526"
server_key_iv: "ce861c7514b59509af0604f60d11d278"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:25.426174 13903 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:54:25.440132 13918 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:25.441663 13903 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:25.441838 13903 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "3b71fb20372346c6a13f6a341a6fff7f"
format_stamp: "Formatted at 2025-10-27 14:54:25 on dist-test-slave-hxxt"
server_key: "8bb0224daec19ab00d6e0a561b0d5526"
server_key_iv: "ce861c7514b59509af0604f60d11d278"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:25.442047 13903 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:25.473706 13903 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:25.477231 13903 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:25.477485 13903 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:25.478385 13903 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:25.479805 13903 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:25.479895 13903 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:25.479993 13903 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:25.480060 13903 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:25.509833 13903 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44497
I20251027 14:54:25.509881 14030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44497 every 8 connection(s)
I20251027 14:54:25.511174 13903 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:54:25.518329 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 13903
I20251027 14:54:25.518520 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:54:25.518927 416 external_mini_cluster.cc:1468] Setting key a19a086784ebb09a2744207c31277f0c
Oct 27 14:54:25 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576865, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:25.528223 14033 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:25.513759 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:59483 (local address 127.0.104.62:35299)
1027 14:54:25.514158 (+ 399us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:25.514162 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:25.515034 (+ 872us) server_negotiation.cc:408] Connection header received
1027 14:54:25.516566 (+ 1532us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:25.516573 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:25.516664 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:25.516767 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:25.518360 (+ 1593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.519426 (+ 1066us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:25.520663 (+ 1237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.521081 (+ 418us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:25.523635 (+ 2554us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:25.523662 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:25.523665 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:25.523695 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:25.526016 (+ 2321us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.526605 (+ 589us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.526609 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.526611 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.526676 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.526918 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.526921 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.526923 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.527123 (+ 200us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:25.527225 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:25.527825 (+ 600us) server_negotiation.cc:300] Negotiation successful
1027 14:54:25.527995 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":218,"threads_started":1}
I20251027 14:54:25.529110 14034 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:25.514199 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35299 (local address 127.0.104.3:59483)
1027 14:54:25.514872 (+ 673us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:25.514909 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:25.516235 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:25.517016 (+ 781us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:25.517031 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:25.517517 (+ 486us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:25.518168 (+ 651us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:25.518185 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.519588 (+ 1403us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:25.519593 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:25.520525 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:25.520540 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.520784 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:25.521458 (+ 674us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:25.521485 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:25.523453 (+ 1968us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:25.526125 (+ 2672us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:25.526132 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:25.526146 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:25.526510 (+ 364us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:25.526744 (+ 234us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:25.526747 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:25.526750 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:25.526852 (+ 102us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:25.527207 (+ 355us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:25.527212 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:25.527486 (+ 274us) client_negotiation.cc:769] Sending connection context
1027 14:54:25.527764 (+ 278us) client_negotiation.cc:240] Negotiation successful
1027 14:54:25.528089 (+ 325us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":289,"threads_started":1}
I20251027 14:54:25.530336 14031 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35299
I20251027 14:54:25.530700 14031 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:25.531431 14031 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:25.532902 13536 ts_manager.cc:194] Registered new tserver with Master: 3b71fb20372346c6a13f6a341a6fff7f (127.0.104.3:44497)
I20251027 14:54:25.533794 13536 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:59483
I20251027 14:54:25.534368 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:54:25.536684 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 13617
Oct 27 14:54:25 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:54:25.551476 14036 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:25.537547 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53870 (local address 127.0.104.1:39393)
1027 14:54:25.538000 (+ 453us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:25.538006 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:25.538031 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:54:25.538090 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:25.538096 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:25.538297 (+ 201us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:25.538440 (+ 143us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:25.540006 (+ 1566us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.541299 (+ 1293us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:25.542519 (+ 1220us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.542923 (+ 404us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:25.546455 (+ 3532us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:25.546522 (+ 67us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:25.546537 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:25.546575 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:25.549189 (+ 2614us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.549770 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.549777 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.549779 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.549863 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.550099 (+ 236us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.550102 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.550104 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.550383 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:25.550602 (+ 219us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:25.550898 (+ 296us) server_negotiation.cc:300] Negotiation successful
1027 14:54:25.551111 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":337,"thread_start_us":227,"threads_started":1}
I20251027 14:54:25.552729 13745 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:25.635648 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 13617
I20251027 14:54:25.648945 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 13758
Oct 27 14:54:25 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:54:25.663365 14038 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:25.649866 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56574 (local address 127.0.104.2:32981)
1027 14:54:25.650373 (+ 507us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:25.650380 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:25.650404 (+ 24us) server_negotiation.cc:408] Connection header received
1027 14:54:25.650468 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:25.650475 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:25.650689 (+ 214us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:25.650839 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:25.652187 (+ 1348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.653430 (+ 1243us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:25.654619 (+ 1189us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.655001 (+ 382us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:25.658424 (+ 3423us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:25.658486 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:25.658502 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:25.658555 (+ 53us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:25.661269 (+ 2714us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.661837 (+ 568us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.661844 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.661846 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.661922 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.662148 (+ 226us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.662151 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.662153 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.662404 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:25.662620 (+ 216us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:25.662818 (+ 198us) server_negotiation.cc:300] Negotiation successful
1027 14:54:25.663033 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":390,"thread_start_us":279,"threads_started":1}
I20251027 14:54:25.664673 13890 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:25.745859 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 13758
I20251027 14:54:25.758432 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 13903
Oct 27 14:54:25 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:54:25.773259 14040 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:25.759338 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43722 (local address 127.0.104.3:44497)
1027 14:54:25.759942 (+ 604us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:25.759949 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:25.759975 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:54:25.760030 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:25.760036 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:25.760239 (+ 203us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:25.760421 (+ 182us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:25.761752 (+ 1331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.762967 (+ 1215us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:25.764153 (+ 1186us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:25.764582 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:25.768132 (+ 3550us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:25.768198 (+ 66us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:25.768214 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:25.768248 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:25.771102 (+ 2854us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.771704 (+ 602us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.771712 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.771714 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.771779 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:25.771999 (+ 220us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:25.772002 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:25.772004 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:25.772291 (+ 287us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:25.772550 (+ 259us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:25.772762 (+ 212us) server_negotiation.cc:300] Negotiation successful
1027 14:54:25.772971 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":486,"thread_start_us":345,"threads_started":1}
I20251027 14:54:25.774492 14026 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:25.855825 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 13903
I20251027 14:54:25.868357 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 13504
I20251027 14:54:25.869670 13566 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:25.974140 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 13504
I20251027 14:54:25.992349 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35299
--webserver_interface=127.0.104.62
--webserver_port=35649
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35299
--ranger_config_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-client
--trusted_user_acl=test-admin
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:26.306514 14043 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:26.306810 14043 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:26.306936 14043 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:26.314949 14043 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:26.315052 14043 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:26.315126 14043 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:26.315189 14043 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:26.315277 14043 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:26.325893 14043 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms
--trusted_user_acl=<redacted>
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35299
--ranger_config_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ranger-client
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35299
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=35649
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:26.327457 14043 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:26.328831 14043 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:26.336791 14048 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:26.337033 14049 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:26.337286 14051 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:26.338132 14043 server_base.cc:1047] running on GCE node
I20251027 14:54:26.338878 14043 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:26.340243 14043 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:26.341461 14043 hybrid_clock.cc:648] HybridClock initialized: now 1761576866341397 us; error 55 us; skew 500 ppm
I20251027 14:54:26.341766 14043 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:26 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:26.344873 14043 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:26.346464 14043 webserver.cc:492] Webserver started at http://127.0.104.62:35649/ using document root <none> and password file <none>
I20251027 14:54:26.347157 14043 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:26.347257 14043 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 27 14:54:26 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:26.416878 14043 fs_manager.cc:714] Time spent opening directory manager: real 0.016s user 0.006s sys 0.006s
I20251027 14:54:26.419945 14059 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:26.421504 14043 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:26.421690 14043 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11"
format_stamp: "Formatted at 2025-10-27 14:54:22 on dist-test-slave-hxxt"
server_key: "4f6b98dc1dceea29a91cdce486872394"
server_key_iv: "90434c41c55a3cb0fb39f3c3ad1d5a77"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:26.422456 14043 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:26.434067 14043 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:26.437546 14043 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:26.437809 14043 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:26.456696 14043 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35299
I20251027 14:54:26.456714 14110 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35299 every 8 connection(s)
I20251027 14:54:26.458218 14043 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:26.463536 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14043
I20251027 14:54:26.464370 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:39393
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:35299
--webserver_port=37719
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:26.465423 14111 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:54:26.468601 14111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Bootstrap starting.
I20251027 14:54:26.485630 14111 log.cc:826] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:26.494776 14111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:54:26.495270 14111 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Bootstrap complete.
I20251027 14:54:26.501583 14111 raft_consensus.cc:359] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:26.502175 14111 raft_consensus.cc:740] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: eb5cb0e1b8784b0a93f0f708dcc74f11, State: Initialized, Role: FOLLOWER
I20251027 14:54:26.502786 14111 consensus_queue.cc:260] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:26.502974 14111 raft_consensus.cc:399] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:54:26.503082 14111 raft_consensus.cc:493] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:54:26.503230 14111 raft_consensus.cc:3060] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 1 FOLLOWER]: Advancing to term 2
I20251027 14:54:26.506719 14111 raft_consensus.cc:515] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:26.507189 14111 leader_election.cc:304] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: eb5cb0e1b8784b0a93f0f708dcc74f11; no voters:
I20251027 14:54:26.507772 14111 leader_election.cc:290] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251027 14:54:26.507906 14115 raft_consensus.cc:2804] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 14:54:26.508968 14115 raft_consensus.cc:697] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [term 2 LEADER]: Becoming Leader. State: Replica: eb5cb0e1b8784b0a93f0f708dcc74f11, State: Running, Role: LEADER
I20251027 14:54:26.509527 14115 consensus_queue.cc:237] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } }
I20251027 14:54:26.509840 14111 sys_catalog.cc:565] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:26.511936 14116 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } } }
I20251027 14:54:26.512236 14116 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:26.517234 14117 sys_catalog.cc:455] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader eb5cb0e1b8784b0a93f0f708dcc74f11. Latest consensus state: current_term: 2 leader_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "eb5cb0e1b8784b0a93f0f708dcc74f11" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35299 } } }
I20251027 14:54:26.517518 14117 sys_catalog.cc:458] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:26.520524 14120 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:26.526477 14120 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:26.527997 14120 catalog_manager.cc:1269] Loaded cluster ID: 707a8a153fc94aacb4fe560396d48a69
I20251027 14:54:26.528151 14120 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:54:26.532037 14120 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:54:26.533465 14111 ranger_client.cc:318] Using overwritten properties file: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/master-0/logs/kudu-ranger-subprocess-log4j2.properties
I20251027 14:54:26.533624 14120 catalog_manager.cc:6033] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Loaded TSK: 0
I20251027 14:54:26.535346 14120 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251027 14:54:26.867674 14113 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:26.868265 14113 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:26.868515 14113 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:26.880913 14113 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:26.881032 14113 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:26.881240 14113 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:54:26.894977 14113 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:39393
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=37719
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:26.896620 14113 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:26.897974 14113 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:26.935417 14148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:26.940366 14149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:26.950801 14151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:26.951097 14113 server_base.cc:1047] running on GCE node
I20251027 14:54:26.952159 14113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:26.957947 14113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:26.959450 14113 hybrid_clock.cc:648] HybridClock initialized: now 1761576866959354 us; error 83 us; skew 500 ppm
I20251027 14:54:26.959916 14113 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:26 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:26.964565 14113 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:54:26.966833 14113 webserver.cc:492] Webserver started at http://127.0.104.1:37719/ using document root <none> and password file <none>
I20251027 14:54:26.967901 14113 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:26.968037 14113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 27 14:54:26 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:26.995291 14113 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20251027 14:54:26.999169 14159 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:27.001539 14113 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20251027 14:54:27.001798 14113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "aa4dbca8af074b39a37447f902c50f79"
format_stamp: "Formatted at 2025-10-27 14:54:23 on dist-test-slave-hxxt"
server_key: "8c36fe4a77ab21d9b614725c43320441"
server_key_iv: "1766de88e0fe3939e9a8ae80e6461412"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:27.002879 14113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:27.075945 14113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:27.081375 14113 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:27.081756 14113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:27.096784 14113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:27.105532 14113 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:27.105731 14113 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251027 14:54:27.105929 14113 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:27.105990 14113 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:27.170363 14113 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:39393
I20251027 14:54:27.170384 14271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:39393 every 8 connection(s)
I20251027 14:54:27.172451 14113 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:54:27.173229 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14113
I20251027 14:54:27.180523 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:32981
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:35299
--webserver_port=34887
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:54:27 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576866, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:27.214982 14275 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:27.177630 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:53313 (local address 127.0.104.62:35299)
1027 14:54:27.189651 (+ 12021us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:27.189676 (+ 25us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:27.190601 (+ 925us) server_negotiation.cc:408] Connection header received
1027 14:54:27.192716 (+ 2115us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:27.192737 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:27.193277 (+ 540us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:27.193836 (+ 559us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:27.196085 (+ 2249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.198061 (+ 1976us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:27.200151 (+ 2090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.200709 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:27.204182 (+ 3473us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:27.204268 (+ 86us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:27.204303 (+ 35us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:27.204357 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:27.208460 (+ 4103us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:27.209525 (+ 1065us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:27.209536 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:27.209545 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:27.209694 (+ 149us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:27.210156 (+ 462us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:27.210162 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:27.210165 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:27.210787 (+ 622us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:27.210956 (+ 169us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:27.213237 (+ 2281us) server_negotiation.cc:300] Negotiation successful
1027 14:54:27.213671 (+ 434us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11724,"thread_start_us":11956,"threads_started":1}
I20251027 14:54:27.216166 14274 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:27.177545 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35299 (local address 127.0.104.1:53313)
1027 14:54:27.190366 (+ 12821us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:27.190416 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:27.191951 (+ 1535us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:27.194048 (+ 2097us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:27.194061 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:27.194777 (+ 716us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:27.195798 (+ 1021us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:27.195830 (+ 32us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.198289 (+ 2459us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:27.198307 (+ 18us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:27.199417 (+ 1110us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:27.199433 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.199713 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:27.201062 (+ 1349us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:27.201098 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:27.203841 (+ 2743us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:27.208685 (+ 4844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:27.208695 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:27.208715 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:27.209325 (+ 610us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:27.209839 (+ 514us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:27.209845 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:27.209849 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:27.210026 (+ 177us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:27.211070 (+ 1044us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:27.211080 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:27.211484 (+ 404us) client_negotiation.cc:769] Sending connection context
1027 14:54:27.211853 (+ 369us) client_negotiation.cc:240] Negotiation successful
1027 14:54:27.212354 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12555,"thread_start_us":12348,"threads_started":1}
I20251027 14:54:27.220486 14272 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35299
I20251027 14:54:27.220832 14272 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:27.221784 14272 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
W20251027 14:54:27.225514 14272 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:35299 (0 consecutive failures): Service unavailable: Catalog manager is not initialized. State: Starting
I20251027 14:54:27.225740 14272 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:27.226105 14272 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:27.227483 14272 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:27.227895 14272 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
W20251027 14:54:27.229090 14272 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251027 14:54:27.587800 14276 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:27.588097 14276 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:27.588228 14276 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:27.596482 14276 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:27.596596 14276 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:27.596815 14276 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:54:27.607769 14276 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:32981
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=34887
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:27.609447 14276 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:27.610845 14276 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:27.628901 14291 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:27.629565 14290 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:27.633715 14293 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:27.634449 14276 server_base.cc:1047] running on GCE node
I20251027 14:54:27.635392 14276 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:27.644332 14276 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:27.647872 14276 hybrid_clock.cc:648] HybridClock initialized: now 1761576867647698 us; error 148 us; skew 500 ppm
I20251027 14:54:27.648413 14276 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:27 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:27.653147 14276 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:54:27.667274 14276 webserver.cc:492] Webserver started at http://127.0.104.2:34887/ using document root <none> and password file <none>
I20251027 14:54:27.668255 14276 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:27.668404 14276 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 27 14:54:27 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:27.698784 14276 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:54:27.701867 14303 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:27.703329 14276 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:54:27.703508 14276 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "e081745b922a44bca5e1093d6adc72ea"
format_stamp: "Formatted at 2025-10-27 14:54:24 on dist-test-slave-hxxt"
server_key: "a82d07deaef7eff175c6c998d29d553f"
server_key_iv: "50d4ac6ef19907be3a75db9d2f49169c"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:27.704358 14276 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:27.736304 14276 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:27.743789 14276 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:27.744076 14276 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:27.745461 14276 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
Oct 27 14:54:27 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1761576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:27.747566 14276 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:27.747644 14276 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:27.747789 14276 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:27.747843 14276 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:27.816500 14276 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:32981
I20251027 14:54:27.816512 14415 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:32981 every 8 connection(s)
I20251027 14:54:27.818485 14276 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:54:27.828856 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14276
I20251027 14:54:27.829748 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--encryption_key_provider=ranger-kms
--encryption_cluster_key_name=kuduclusterkey
--ranger_kms_url=127.0.104.20:50689/kms
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:44497
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:35299
--webserver_port=39043
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:33853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:54:27 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:27.842121 14418 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:27.821573 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:37417 (local address 127.0.104.62:35299)
1027 14:54:27.821979 (+ 406us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:27.821984 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:27.823171 (+ 1187us) server_negotiation.cc:408] Connection header received
1027 14:54:27.824721 (+ 1550us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:27.824728 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:27.824813 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:27.824919 (+ 106us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:27.826552 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.827619 (+ 1067us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:27.830057 (+ 2438us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.830560 (+ 503us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:27.834809 (+ 4249us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:27.834860 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:27.834865 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:27.834906 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:27.838314 (+ 3408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:27.839325 (+ 1011us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:27.839332 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:27.839336 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:27.839429 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:27.839879 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:27.839885 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:27.839888 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:27.840188 (+ 300us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:27.840336 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:27.841494 (+ 1158us) server_negotiation.cc:300] Negotiation successful
1027 14:54:27.841761 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":312,"thread_start_us":207,"threads_started":1}
I20251027 14:54:27.844652 14419 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:27.822225 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35299 (local address 127.0.104.2:37417)
1027 14:54:27.822976 (+ 751us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:27.823018 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:27.824433 (+ 1415us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:27.825183 (+ 750us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:27.825193 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:27.825665 (+ 472us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:27.826362 (+ 697us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:27.826380 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.827813 (+ 1433us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:27.827819 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:27.829866 (+ 2047us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:27.829883 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:27.830923 (+ 1040us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:27.831741 (+ 818us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:27.831784 (+ 43us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:27.834518 (+ 2734us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:27.838542 (+ 4024us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:27.838552 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:27.838574 (+ 22us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:27.839164 (+ 590us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:27.839576 (+ 412us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:27.839581 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:27.839585 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:27.839754 (+ 169us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:27.840515 (+ 761us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:27.840524 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:27.840950 (+ 426us) client_negotiation.cc:769] Sending connection context
1027 14:54:27.842450 (+ 1500us) client_negotiation.cc:240] Negotiation successful
1027 14:54:27.842947 (+ 497us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":513,"thread_start_us":321,"threads_started":1}
I20251027 14:54:27.846810 14416 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35299
I20251027 14:54:27.847229 14416 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:27.848225 14416 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:27.850533 14076 ts_manager.cc:194] Registered new tserver with Master: e081745b922a44bca5e1093d6adc72ea (127.0.104.2:32981)
I20251027 14:54:27.853020 14076 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:37417
Oct 27 14:54:28 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:28.229259 14272 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:28.229830 14272 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:28.231244 14076 ts_manager.cc:194] Registered new tserver with Master: aa4dbca8af074b39a37447f902c50f79 (127.0.104.1:39393)
I20251027 14:54:28.232340 14076 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:53313
W20251027 14:54:28.248559 14420 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:28.248948 14420 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:28.249115 14420 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:28.262054 14420 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:28.262189 14420 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:28.262408 14420 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:54:28.279043 14420 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:33853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--encryption_cluster_key_name=kuduclusterkey
--encryption_key_provider=ranger-kms
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
--ranger_kms_url=invalid.host:1234/kms,127.0.104.20:50689/kms
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:44497
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=39043
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:35299
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:28.281327 14420 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:28.283149 14420 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:28.294423 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:28.295236 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:28.296906 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:28.297721 14420 server_base.cc:1047] running on GCE node
I20251027 14:54:28.298435 14420 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:28.299610 14420 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:28.300923 14420 hybrid_clock.cc:648] HybridClock initialized: now 1761576868300838 us; error 53 us; skew 500 ppm
I20251027 14:54:28.301312 14420 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:28 dist-test-slave-hxxt krb5kdc[13033](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:28.305605 14420 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:54:28.307713 14420 webserver.cc:492] Webserver started at http://127.0.104.3:39043/ using document root <none> and password file <none>
I20251027 14:54:28.308717 14420 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:28.308830 14420 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
Oct 27 14:54:28 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:28.358976 14420 fs_manager.cc:714] Time spent opening directory manager: real 0.022s user 0.011s sys 0.005s
I20251027 14:54:28.375974 14437 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:28.378158 14420 fs_manager.cc:730] Time spent opening block manager: real 0.017s user 0.015s sys 0.000s
I20251027 14:54:28.378453 14420 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "3b71fb20372346c6a13f6a341a6fff7f"
format_stamp: "Formatted at 2025-10-27 14:54:25 on dist-test-slave-hxxt"
server_key: "8bb0224daec19ab00d6e0a561b0d5526"
server_key_iv: "ce861c7514b59509af0604f60d11d278"
server_key_version: "kuduclusterkey@0"
I20251027 14:54:28.379616 14420 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:28.441566 14420 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:28.446851 14420 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:28.447250 14420 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:28.461277 14420 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
Oct 27 14:54:28 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576867, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for HTTP/127.0.104.20@KRBTEST.COM
I20251027 14:54:28.470508 14420 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:28.470692 14420 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251027 14:54:28.470855 14420 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:28.470953 14420 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:28.714877 14420 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:44497
I20251027 14:54:28.715067 14550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:44497 every 8 connection(s)
I20251027 14:54:28.717080 14420 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:54:28.726958 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14420
Oct 27 14:54:28 dist-test-slave-hxxt krb5kdc[13033](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576868, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:28.774235 14553 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.726792 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35299 (local address 127.0.104.3:41259)
1027 14:54:28.728288 (+ 1496us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:28.728337 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:28.730358 (+ 2021us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:28.754700 (+ 24342us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:28.754715 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:28.755495 (+ 780us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:28.756865 (+ 1370us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.756902 (+ 37us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.759195 (+ 2293us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.759202 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:28.760437 (+ 1235us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.760455 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.760737 (+ 282us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.761558 (+ 821us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:28.761593 (+ 35us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:28.764320 (+ 2727us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:28.769154 (+ 4834us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:28.769166 (+ 12us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:28.769187 (+ 21us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:28.769767 (+ 580us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:28.770154 (+ 387us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:28.770160 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:28.770163 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:28.770345 (+ 182us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:28.771005 (+ 660us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:28.771014 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:28.771456 (+ 442us) client_negotiation.cc:769] Sending connection context
1027 14:54:28.772168 (+ 712us) client_negotiation.cc:240] Negotiation successful
1027 14:54:28.772700 (+ 532us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1162,"thread_start_us":4630,"threads_started":1}
I20251027 14:54:28.777401 14556 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.727578 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:41259 (local address 127.0.104.62:35299)
1027 14:54:28.753253 (+ 25675us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.753261 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.753293 (+ 32us) server_negotiation.cc:408] Connection header received
1027 14:54:28.753369 (+ 76us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.753375 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.753463 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.757118 (+ 3655us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:28.757449 (+ 331us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.758994 (+ 1545us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.761207 (+ 2213us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.763345 (+ 2138us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.764805 (+ 1460us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:28.764869 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:28.764875 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:28.764914 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:28.768920 (+ 4006us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:28.769930 (+ 1010us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:28.769935 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:28.769951 (+ 16us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:28.770033 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:28.770485 (+ 452us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:28.770490 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:28.770494 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:28.770815 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:28.776552 (+ 5737us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.776713 (+ 161us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.776992 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":25532,"thread_start_us":37012,"threads_started":1}
I20251027 14:54:28.779978 14551 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35299
I20251027 14:54:28.780509 14551 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:28.781679 14551 heartbeater.cc:507] Master 127.0.104.62:35299 requested a full tablet report, sending...
I20251027 14:54:28.783553 14076 ts_manager.cc:194] Registered new tserver with Master: 3b71fb20372346c6a13f6a341a6fff7f (127.0.104.3:44497)
I20251027 14:54:28.784754 14076 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:41259
I20251027 14:54:28.785276 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.728436 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50162 (local address 127.0.104.62:35299)
1027 14:54:28.764900 (+ 36464us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.764906 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.764924 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:28.764984 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.764990 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.765071 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.765192 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:28.767988 (+ 2796us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.769687 (+ 1699us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.771837 (+ 2150us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.772588 (+ 751us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.777753 (+ 5165us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:28.777824 (+ 71us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:28.777830 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:28.777866 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:28.782405 (+ 4539us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:28.783288 (+ 883us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:28.783294 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:28.783298 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:28.783374 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:28.783750 (+ 376us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:28.783757 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:28.783760 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:28.784036 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:28.784314 (+ 278us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.784617 (+ 303us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.784876 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36334,"thread_start_us":32905,"threads_started":1}
I20251027 14:54:28.789213 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:54:28.805373 14554 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.792656 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50166 (local address 127.0.104.62:35299)
1027 14:54:28.792831 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.792837 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.793268 (+ 431us) server_negotiation.cc:408] Connection header received
1027 14:54:28.793427 (+ 159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.793433 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.793516 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.793623 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:28.795395 (+ 1772us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.796361 (+ 966us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.798019 (+ 1658us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.798471 (+ 452us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.800457 (+ 1986us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:28.800489 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:28.800493 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:28.800528 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:28.802873 (+ 2345us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:28.803592 (+ 719us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:28.803597 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:28.803599 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:28.803685 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:28.803989 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:28.804001 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:28.804003 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:28.804242 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:28.804521 (+ 279us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.804821 (+ 300us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.805078 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:54:28.810034 14076 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50166:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:54:28.815153 14076 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:54:28.842161 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.829339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39393 (local address 127.0.0.1:53880)
1027 14:54:28.830518 (+ 1179us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:28.830557 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:28.830753 (+ 196us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:28.831860 (+ 1107us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:28.831869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:28.831905 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:28.834072 (+ 2167us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.834089 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.837947 (+ 3858us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.837954 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:28.839857 (+ 1903us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.839873 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.840038 (+ 165us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.840118 (+ 80us) client_negotiation.cc:769] Sending connection context
1027 14:54:28.840289 (+ 171us) client_negotiation.cc:240] Negotiation successful
1027 14:54:28.841713 (+ 1424us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":954,"thread_start_us":1406,"threads_started":1}
I20251027 14:54:28.842492 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.829621 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53880 (local address 127.0.104.1:39393)
1027 14:54:28.830689 (+ 1068us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.830696 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.830721 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:54:28.831024 (+ 303us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.831034 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.831291 (+ 257us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.831496 (+ 205us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:28.834309 (+ 2813us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.837679 (+ 3370us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.840638 (+ 2959us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.841757 (+ 1119us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.841915 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.842046 (+ 131us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.842215 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":946,"thread_start_us":846,"threads_started":1}
I20251027 14:54:28.847184 14575 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.830227 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:44497 (local address 127.0.0.1:43736)
1027 14:54:28.839823 (+ 9596us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:28.839842 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:28.839956 (+ 114us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:28.841582 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:28.841588 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:28.841616 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:28.842423 (+ 807us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.842438 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.844590 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.844596 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:28.846466 (+ 1870us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.846477 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.846646 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.846671 (+ 25us) client_negotiation.cc:769] Sending connection context
1027 14:54:28.846734 (+ 63us) client_negotiation.cc:240] Negotiation successful
1027 14:54:28.846854 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":9473,"thread_start_us":9331,"threads_started":1}
I20251027 14:54:28.847692 14207 tablet_service.cc:1505] Processing CreateTablet for tablet bb024dd4eb7d4ecc8a396863e29d86fa (DEFAULT_TABLE table=test-table [id=900df0bcf9a64430ad23a59e4f7109b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:28.848408 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.830883 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43736 (local address 127.0.104.3:44497)
1027 14:54:28.840610 (+ 9727us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.840618 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.840651 (+ 33us) server_negotiation.cc:408] Connection header received
1027 14:54:28.840747 (+ 96us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.840756 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.841045 (+ 289us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.841258 (+ 213us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:28.842641 (+ 1383us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.844413 (+ 1772us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.846642 (+ 2229us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.847721 (+ 1079us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.847854 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.847969 (+ 115us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.848135 (+ 166us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9602,"thread_start_us":9448,"threads_started":1}
I20251027 14:54:28.848443 14573 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.831536 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:32981 (local address 127.0.0.1:56578)
1027 14:54:28.842638 (+ 11102us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:28.842658 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:28.842862 (+ 204us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:28.843742 (+ 880us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:28.843745 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:28.843765 (+ 20us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:28.844032 (+ 267us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.844039 (+ 7us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.846159 (+ 2120us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.846165 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:28.847823 (+ 1658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.847835 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.847959 (+ 124us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.847982 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:54:28.848043 (+ 61us) client_negotiation.cc:240] Negotiation successful
1027 14:54:28.848156 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11013,"thread_start_us":11478,"threads_started":1}
I20251027 14:54:28.849480 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.831882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56578 (local address 127.0.104.2:32981)
1027 14:54:28.840164 (+ 8282us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.840174 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.842760 (+ 2586us) server_negotiation.cc:408] Connection header received
1027 14:54:28.843056 (+ 296us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.843064 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.843330 (+ 266us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.843486 (+ 156us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:28.844195 (+ 709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.845975 (+ 1780us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.847969 (+ 1994us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.848931 (+ 962us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.849033 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.849124 (+ 91us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.849241 (+ 117us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":8143,"thread_start_us":8025,"threads_started":1}
I20251027 14:54:28.850066 14207 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb024dd4eb7d4ecc8a396863e29d86fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:28.852094 14486 tablet_service.cc:1505] Processing CreateTablet for tablet bb024dd4eb7d4ecc8a396863e29d86fa (DEFAULT_TABLE table=test-table [id=900df0bcf9a64430ad23a59e4f7109b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:28.853600 14486 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb024dd4eb7d4ecc8a396863e29d86fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:28.853586 14351 tablet_service.cc:1505] Processing CreateTablet for tablet bb024dd4eb7d4ecc8a396863e29d86fa (DEFAULT_TABLE table=test-table [id=900df0bcf9a64430ad23a59e4f7109b4]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:28.855015 14351 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bb024dd4eb7d4ecc8a396863e29d86fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:28.857745 14416 heartbeater.cc:499] Master 127.0.104.62:35299 was elected leader, sending a full tablet report...
I20251027 14:54:28.864666 14579 tablet_bootstrap.cc:492] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Bootstrap starting.
I20251027 14:54:28.865653 14580 tablet_bootstrap.cc:492] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Bootstrap starting.
I20251027 14:54:28.868924 14581 tablet_bootstrap.cc:492] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Bootstrap starting.
I20251027 14:54:28.869272 14580 tablet_bootstrap.cc:654] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:28.868350 14579 tablet_bootstrap.cc:654] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:28.870477 14580 log.cc:826] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:28.870774 14579 log.cc:826] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:28.872541 14581 tablet_bootstrap.cc:654] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:28.873577 14579 tablet_bootstrap.cc:492] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: No bootstrap required, opened a new log
I20251027 14:54:28.873808 14580 tablet_bootstrap.cc:492] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: No bootstrap required, opened a new log
I20251027 14:54:28.873832 14579 ts_tablet_manager.cc:1403] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20251027 14:54:28.873798 14581 log.cc:826] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:28.874094 14580 ts_tablet_manager.cc:1403] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20251027 14:54:28.877151 14581 tablet_bootstrap.cc:492] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: No bootstrap required, opened a new log
I20251027 14:54:28.877451 14581 ts_tablet_manager.cc:1403] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Time spent bootstrapping tablet: real 0.009s user 0.004s sys 0.003s
I20251027 14:54:28.879132 14579 raft_consensus.cc:359] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.879877 14579 raft_consensus.cc:385] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:28.880046 14579 raft_consensus.cc:740] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3b71fb20372346c6a13f6a341a6fff7f, State: Initialized, Role: FOLLOWER
I20251027 14:54:28.880847 14579 consensus_queue.cc:260] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.880733 14580 raft_consensus.cc:359] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.881143 14580 raft_consensus.cc:385] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:28.881269 14580 raft_consensus.cc:740] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aa4dbca8af074b39a37447f902c50f79, State: Initialized, Role: FOLLOWER
I20251027 14:54:28.881884 14551 heartbeater.cc:499] Master 127.0.104.62:35299 was elected leader, sending a full tablet report...
I20251027 14:54:28.881911 14580 consensus_queue.cc:260] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.882503 14579 ts_tablet_manager.cc:1434] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Time spent starting tablet: real 0.009s user 0.009s sys 0.000s
I20251027 14:54:28.883364 14272 heartbeater.cc:499] Master 127.0.104.62:35299 was elected leader, sending a full tablet report...
I20251027 14:54:28.883908 14580 ts_tablet_manager.cc:1434] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20251027 14:54:28.884536 14581 raft_consensus.cc:359] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.884872 14581 raft_consensus.cc:385] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:28.884979 14581 raft_consensus.cc:740] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e081745b922a44bca5e1093d6adc72ea, State: Initialized, Role: FOLLOWER
I20251027 14:54:28.885984 14581 consensus_queue.cc:260] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.888446 14581 ts_tablet_manager.cc:1434] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Time spent starting tablet: real 0.011s user 0.010s sys 0.001s
I20251027 14:54:28.903281 14587 raft_consensus.cc:493] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:28.903548 14587 raft_consensus.cc:515] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.905548 14587 leader_election.cc:290] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aa4dbca8af074b39a37447f902c50f79 (127.0.104.1:39393), 3b71fb20372346c6a13f6a341a6fff7f (127.0.104.3:44497)
I20251027 14:54:28.911530 14588 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.905849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:44497 (local address 127.0.104.2:45379)
1027 14:54:28.906547 (+ 698us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:28.906568 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:28.906720 (+ 152us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:28.907010 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:28.907015 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:28.907039 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:28.907727 (+ 688us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.907737 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.909209 (+ 1472us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.909213 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:28.910730 (+ 1517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.910746 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.910978 (+ 232us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.911010 (+ 32us) client_negotiation.cc:769] Sending connection context
1027 14:54:28.911082 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:54:28.911217 (+ 135us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":586,"thread_start_us":335,"threads_started":1}
I20251027 14:54:28.911650 14589 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.905909 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:39393 (local address 127.0.104.2:40705)
1027 14:54:28.906636 (+ 727us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:28.906650 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:28.906773 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:28.907073 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:28.907078 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:28.907135 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:28.907784 (+ 649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.907794 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.909282 (+ 1488us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.909288 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:28.910852 (+ 1564us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:28.910865 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.911124 (+ 259us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.911170 (+ 46us) client_negotiation.cc:769] Sending connection context
1027 14:54:28.911244 (+ 74us) client_negotiation.cc:240] Negotiation successful
1027 14:54:28.911357 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":649,"spinlock_wait_cycles":896,"thread_start_us":468,"threads_started":1}
I20251027 14:54:28.912410 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.906048 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:45379 (local address 127.0.104.3:44497)
1027 14:54:28.906229 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.906234 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.906673 (+ 439us) server_negotiation.cc:408] Connection header received
1027 14:54:28.906847 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.906851 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.906910 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.906992 (+ 82us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:28.907867 (+ 875us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.908994 (+ 1127us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.910906 (+ 1912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.911929 (+ 1023us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.911994 (+ 65us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.912068 (+ 74us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.912170 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20251027 14:54:28.912609 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.905831 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:40705 (local address 127.0.104.1:39393)
1027 14:54:28.906009 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.906016 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.906743 (+ 727us) server_negotiation.cc:408] Connection header received
1027 14:54:28.906899 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.906904 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.906967 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.907042 (+ 75us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:28.907926 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.908978 (+ 1052us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.911023 (+ 2045us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.912137 (+ 1114us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.912189 (+ 52us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.912274 (+ 85us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.912370 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:54:28.913321 14506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb024dd4eb7d4ecc8a396863e29d86fa" candidate_uuid: "e081745b922a44bca5e1093d6adc72ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b71fb20372346c6a13f6a341a6fff7f" is_pre_election: true
I20251027 14:54:28.913352 14227 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb024dd4eb7d4ecc8a396863e29d86fa" candidate_uuid: "e081745b922a44bca5e1093d6adc72ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4dbca8af074b39a37447f902c50f79" is_pre_election: true
I20251027 14:54:28.913762 14227 raft_consensus.cc:2468] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e081745b922a44bca5e1093d6adc72ea in term 0.
I20251027 14:54:28.913762 14506 raft_consensus.cc:2468] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e081745b922a44bca5e1093d6adc72ea in term 0.
I20251027 14:54:28.914354 14306 leader_election.cc:304] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3b71fb20372346c6a13f6a341a6fff7f, e081745b922a44bca5e1093d6adc72ea; no voters:
I20251027 14:54:28.914698 14587 raft_consensus.cc:2804] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:54:28.914860 14587 raft_consensus.cc:493] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:54:28.914976 14587 raft_consensus.cc:3060] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:28.917264 14587 raft_consensus.cc:515] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
I20251027 14:54:28.917807 14587 leader_election.cc:290] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [CANDIDATE]: Term 1 election: Requested vote from peers aa4dbca8af074b39a37447f902c50f79 (127.0.104.1:39393), 3b71fb20372346c6a13f6a341a6fff7f (127.0.104.3:44497)
I20251027 14:54:28.918282 14227 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb024dd4eb7d4ecc8a396863e29d86fa" candidate_uuid: "e081745b922a44bca5e1093d6adc72ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aa4dbca8af074b39a37447f902c50f79"
I20251027 14:54:28.918323 14506 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "bb024dd4eb7d4ecc8a396863e29d86fa" candidate_uuid: "e081745b922a44bca5e1093d6adc72ea" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3b71fb20372346c6a13f6a341a6fff7f"
I20251027 14:54:28.918489 14227 raft_consensus.cc:3060] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:28.918489 14506 raft_consensus.cc:3060] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:28.921979 14506 raft_consensus.cc:2468] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e081745b922a44bca5e1093d6adc72ea in term 1.
I20251027 14:54:28.922013 14227 raft_consensus.cc:2468] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e081745b922a44bca5e1093d6adc72ea in term 1.
I20251027 14:54:28.922482 14306 leader_election.cc:304] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 3b71fb20372346c6a13f6a341a6fff7f, aa4dbca8af074b39a37447f902c50f79, e081745b922a44bca5e1093d6adc72ea; no voters:
I20251027 14:54:28.922813 14587 raft_consensus.cc:2804] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:28.923254 14587 raft_consensus.cc:697] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 1 LEADER]: Becoming Leader. State: Replica: e081745b922a44bca5e1093d6adc72ea, State: Running, Role: LEADER
I20251027 14:54:28.923755 14587 consensus_queue.cc:237] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } }
W20251027 14:54:28.928033 14273 tablet.cc:2378] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:54:28.928678 14076 catalog_manager.cc:5649] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea reported cstate change: term changed from 0 to 1, leader changed from <none> to e081745b922a44bca5e1093d6adc72ea (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e081745b922a44bca5e1093d6adc72ea" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e081745b922a44bca5e1093d6adc72ea" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 32981 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 } health_report { overall_health: UNKNOWN } } }
I20251027 14:54:28.964730 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:28.959856 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56580 (local address 127.0.104.2:32981)
1027 14:54:28.960041 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:28.960048 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:28.960066 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:28.960156 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:28.960160 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:28.960246 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:28.960395 (+ 149us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:54:28.960955 (+ 560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.961841 (+ 886us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:28.963085 (+ 1244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:28.963456 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:28.963604 (+ 148us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:54:28.964120 (+ 516us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:54:28.964249 (+ 129us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:28.964365 (+ 116us) server_negotiation.cc:300] Negotiation successful
1027 14:54:28.964492 (+ 127us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:54:28.972553 14506 raft_consensus.cc:1275] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 1 FOLLOWER]: Refusing update from remote peer e081745b922a44bca5e1093d6adc72ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:28.972535 14227 raft_consensus.cc:1275] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 1 FOLLOWER]: Refusing update from remote peer e081745b922a44bca5e1093d6adc72ea: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:28.973479 14590 consensus_queue.cc:1048] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "3b71fb20372346c6a13f6a341a6fff7f" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 44497 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:28.973796 14587 consensus_queue.cc:1048] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [LEADER]: Connected to new peer: Peer: permanent_uuid: "aa4dbca8af074b39a37447f902c50f79" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 39393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20251027 14:54:28.977993 14552 tablet.cc:2378] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:54:29.000051 14076 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50166:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:54:29.000295 14076 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50166:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:54:29.003701 14076 catalog_manager.cc:5936] T 00000000000000000000000000000000 P eb5cb0e1b8784b0a93f0f708dcc74f11: Sending DeleteTablet for 3 replicas of tablet bb024dd4eb7d4ecc8a396863e29d86fa
I20251027 14:54:29.004925 14486 tablet_service.cc:1552] Processing DeleteTablet for tablet bb024dd4eb7d4ecc8a396863e29d86fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:28 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:43736
I20251027 14:54:29.004899 14351 tablet_service.cc:1552] Processing DeleteTablet for tablet bb024dd4eb7d4ecc8a396863e29d86fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:28 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:56578
I20251027 14:54:29.005771 14602 tablet_replica.cc:333] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: stopping tablet replica
I20251027 14:54:29.005867 14207 tablet_service.cc:1552] Processing DeleteTablet for tablet bb024dd4eb7d4ecc8a396863e29d86fa with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:28 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:53880
I20251027 14:54:29.006193 14602 raft_consensus.cc:2243] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:54:29.006690 14602 raft_consensus.cc:2272] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:29.007483 14603 tablet_replica.cc:333] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: stopping tablet replica
I20251027 14:54:29.007977 14603 raft_consensus.cc:2243] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:54:29.008791 14603 raft_consensus.cc:2272] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:29.009730 14604 tablet_replica.cc:333] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: stopping tablet replica
I20251027 14:54:29.010142 14604 raft_consensus.cc:2243] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:54:29.010542 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14113
I20251027 14:54:29.010634 14604 raft_consensus.cc:2272] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:29.011063 14603 ts_tablet_manager.cc:1916] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:29.013482 14604 ts_tablet_manager.cc:1916] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:29.015326 14602 ts_tablet_manager.cc:1916] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:29.020987 14604 ts_tablet_manager.cc:1929] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:29.021090 14602 ts_tablet_manager.cc:1929] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:29.021253 14604 log.cc:1199] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/bb024dd4eb7d4ecc8a396863e29d86fa
I20251027 14:54:29.021747 14604 ts_tablet_manager.cc:1950] T bb024dd4eb7d4ecc8a396863e29d86fa P aa4dbca8af074b39a37447f902c50f79: Deleting consensus metadata
I20251027 14:54:29.023562 14061 catalog_manager.cc:4985] TS aa4dbca8af074b39a37447f902c50f79 (127.0.104.1:39393): tablet bb024dd4eb7d4ecc8a396863e29d86fa (table test-table [id=900df0bcf9a64430ad23a59e4f7109b4]) successfully deleted
I20251027 14:54:29.026639 14603 ts_tablet_manager.cc:1929] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:29.026845 14603 log.cc:1199] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/bb024dd4eb7d4ecc8a396863e29d86fa
I20251027 14:54:29.027302 14602 log.cc:1199] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/bb024dd4eb7d4ecc8a396863e29d86fa
I20251027 14:54:29.027370 14603 ts_tablet_manager.cc:1950] T bb024dd4eb7d4ecc8a396863e29d86fa P e081745b922a44bca5e1093d6adc72ea: Deleting consensus metadata
I20251027 14:54:29.027837 14602 ts_tablet_manager.cc:1950] T bb024dd4eb7d4ecc8a396863e29d86fa P 3b71fb20372346c6a13f6a341a6fff7f: Deleting consensus metadata
I20251027 14:54:29.029161 14574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:29.011422 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53882 (local address 127.0.104.1:39393)
1027 14:54:29.011743 (+ 321us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:29.011750 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:29.012071 (+ 321us) server_negotiation.cc:408] Connection header received
1027 14:54:29.012160 (+ 89us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:29.012166 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:29.012258 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:29.012360 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:29.015529 (+ 3169us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:29.016889 (+ 1360us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:29.018648 (+ 1759us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:29.019172 (+ 524us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:29.021512 (+ 2340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:29.021571 (+ 59us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:29.021592 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:29.021630 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:29.025671 (+ 4041us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:29.026518 (+ 847us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:29.026529 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:29.026533 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:29.026631 (+ 98us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:29.027193 (+ 562us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:29.027199 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:29.027202 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:29.027592 (+ 390us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:29.027885 (+ 293us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:29.028288 (+ 403us) server_negotiation.cc:300] Negotiation successful
1027 14:54:29.028585 (+ 297us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":210}
I20251027 14:54:29.029117 14061 catalog_manager.cc:4985] TS e081745b922a44bca5e1093d6adc72ea (127.0.104.2:32981): tablet bb024dd4eb7d4ecc8a396863e29d86fa (table test-table [id=900df0bcf9a64430ad23a59e4f7109b4]) successfully deleted
I20251027 14:54:29.030083 14267 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:29.030483 14062 catalog_manager.cc:4985] TS 3b71fb20372346c6a13f6a341a6fff7f (127.0.104.3:44497): tablet bb024dd4eb7d4ecc8a396863e29d86fa (table test-table [id=900df0bcf9a64430ad23a59e4f7109b4]) successfully deleted
I20251027 14:54:29.117894 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14113
I20251027 14:54:29.137737 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14276
I20251027 14:54:29.152463 14578 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:29.138541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56582 (local address 127.0.104.2:32981)
1027 14:54:29.138700 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:29.138706 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:29.138880 (+ 174us) server_negotiation.cc:408] Connection header received
1027 14:54:29.139088 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:29.139093 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:29.139163 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:29.139252 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:29.141202 (+ 1950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:29.142175 (+ 973us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:29.143784 (+ 1609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:29.144191 (+ 407us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:29.146759 (+ 2568us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:29.146819 (+ 60us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:29.146835 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:29.146866 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:29.149964 (+ 3098us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:29.150727 (+ 763us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:29.150734 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:29.150737 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:29.150816 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:29.151131 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:29.151134 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:29.151136 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:29.151404 (+ 268us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:29.151649 (+ 245us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:29.151933 (+ 284us) server_negotiation.cc:300] Negotiation successful
1027 14:54:29.152134 (+ 201us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
I20251027 14:54:29.153218 14411 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:29.260679 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14276
I20251027 14:54:29.275518 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14420
I20251027 14:54:29.287822 14576 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:29.276236 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:43748 (local address 127.0.104.3:44497)
1027 14:54:29.276455 (+ 219us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:29.276460 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:29.276478 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:29.276638 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:29.276645 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:29.276720 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:29.276811 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:29.278188 (+ 1377us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:29.279046 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:29.280271 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:29.280674 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:29.282518 (+ 1844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:29.282562 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:29.282578 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:29.282610 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:29.285553 (+ 2943us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:29.286179 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:29.286186 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:29.286189 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:29.286271 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:29.286554 (+ 283us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:29.286557 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:29.286559 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:29.286850 (+ 291us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:29.287112 (+ 262us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:29.287349 (+ 237us) server_negotiation.cc:300] Negotiation successful
1027 14:54:29.287523 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
I20251027 14:54:29.288463 14546 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:29.375349 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14420
I20251027 14:54:29.388513 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14043
I20251027 14:54:29.389719 14106 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:29.498872 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14043
I20251027 14:54:29.516980 416 mini_ranger_kms.cc:60] Stopping Ranger KMS...
I20251027 14:54:29.852494 416 mini_ranger_kms.cc:62] Stopped Ranger KMS
I20251027 14:54:29.852686 416 mini_ranger.cc:66] Stopping Ranger...
I20251027 14:54:30.188086 416 mini_ranger.cc:68] Stopped Ranger
2025-10-27 14:54:30.188 UTC [13064] LOG: received smart shutdown request
2025-10-27 14:54:30.190 UTC [13064] LOG: background worker "logical replication launcher" (PID 13072) exited with exit code 1
2025-10-27 14:54:30.191 UTC [13067] LOG: shutting down
2025-10-27 14:54:30.193 UTC [13067] LOG: checkpoint starting: shutdown immediate
2025-10-27 14:54:30.953 UTC [13067] LOG: checkpoint complete: wrote 2411 buffers (14.7%); 0 WAL file(s) added, 0 removed, 1 recycled; write=0.033 s, sync=0.722 s, total=0.762 s; sync files=1108, longest=0.005 s, average=0.001 s; distance=12075 kB, estimate=12075 kB; lsn=0/2045540, redo lsn=0/2045540
2025-10-27 14:54:30.960 UTC [13064] LOG: database system is shut down
2025-10-27T14:54:30Z chronyd exiting
[ OK ] SecurityITest.TestEncryptionWithKMSIntegrationMultipleServers (52585 ms)
[ RUN ] SecurityITest.IPKICACert
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:54:31 dist-test-slave-hxxt krb5kdc[14617](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:54:31 dist-test-slave-hxxt krb5kdc[14617](info): set up 2 sockets
Oct 27 14:54:31 dist-test-slave-hxxt krb5kdc[14617](info): commencing operation
krb5kdc: starting...
W20251027 14:54:32.948689 416 mini_kdc.cc:121] Time spent starting KDC: real 1.843s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:54:33 dist-test-slave-hxxt krb5kdc[14617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:54:33Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:54:33Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:33.205942 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41877
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41877,127.0.104.61:38393,127.0.104.60:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:33.527499 14633 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:33.527818 14633 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:33.527961 14633 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:33.536031 14633 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:33.536149 14633 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:33.536231 14633 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:33.536293 14633 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:33.536392 14633 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:33.546965 14633 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41877,127.0.104.61:38393,127.0.104.60:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:41877
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:33.548605 14633 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:33.549954 14633 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:33.557752 14639 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:33.557726 14638 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:33.559079 14641 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:33.559209 14633 server_base.cc:1047] running on GCE node
I20251027 14:54:33.560019 14633 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:33.561406 14633 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:33.562614 14633 hybrid_clock.cc:648] HybridClock initialized: now 1761576873562546 us; error 41 us; skew 500 ppm
I20251027 14:54:33.562923 14633 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:33 dist-test-slave-hxxt krb5kdc[14617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:33.565927 14633 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:33.567437 14633 webserver.cc:492] Webserver started at http://127.0.104.62:45077/ using document root <none> and password file <none>
I20251027 14:54:33.568148 14633 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:33.568238 14633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:33.568637 14633 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:33.571285 14633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "cee2ec5a9da34357be142cb158e21dc2"
format_stamp: "Formatted at 2025-10-27 14:54:33 on dist-test-slave-hxxt"
server_key: "a2486349154281bd6513f13ae00af8f3"
server_key_iv: "53fbb33afa25872b30fa48701c6cecfb"
server_key_version: "encryptionkey@0"
I20251027 14:54:33.572047 14633 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "cee2ec5a9da34357be142cb158e21dc2"
format_stamp: "Formatted at 2025-10-27 14:54:33 on dist-test-slave-hxxt"
server_key: "a2486349154281bd6513f13ae00af8f3"
server_key_iv: "53fbb33afa25872b30fa48701c6cecfb"
server_key_version: "encryptionkey@0"
I20251027 14:54:33.577553 14633 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:54:33.581341 14648 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:33.582855 14633 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:54:33.583038 14633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "cee2ec5a9da34357be142cb158e21dc2"
format_stamp: "Formatted at 2025-10-27 14:54:33 on dist-test-slave-hxxt"
server_key: "a2486349154281bd6513f13ae00af8f3"
server_key_iv: "53fbb33afa25872b30fa48701c6cecfb"
server_key_version: "encryptionkey@0"
I20251027 14:54:33.583258 14633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:33.604341 14633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:33.613874 14633 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:33.614156 14633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:33.633574 14633 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:41877
I20251027 14:54:33.633598 14699 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:41877 every 8 connection(s)
I20251027 14:54:33.635082 14633 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:33.636289 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14633
I20251027 14:54:33.636555 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:54:33.637265 416 external_mini_cluster.cc:1468] Setting key 886249633f68ab974f39db10ca20d2d9
I20251027 14:54:33.640413 14700 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:33.648185 14700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.61@KRBTEST.COM" created.
Oct 27 14:54:33 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:33.669516 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.651216 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48252 (local address 127.0.104.62:41877)
1027 14:54:33.653590 (+ 2374us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:33.653600 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:33.653652 (+ 52us) server_negotiation.cc:408] Connection header received
1027 14:54:33.654308 (+ 656us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:33.654315 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:33.654567 (+ 252us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:33.654772 (+ 205us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:33.656265 (+ 1493us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:33.657460 (+ 1195us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:33.658413 (+ 953us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:33.658977 (+ 564us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:33.662171 (+ 3194us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:33.662249 (+ 78us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:33.662268 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:33.662303 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:33.665580 (+ 3277us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:33.666277 (+ 697us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:33.666286 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:33.666289 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:33.666401 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:33.666716 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:33.666721 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:33.666724 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:33.667321 (+ 597us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:33.667495 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:33.668073 (+ 578us) server_negotiation.cc:300] Negotiation successful
1027 14:54:33.668354 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2205,"spinlock_wait_cycles":22400,"thread_start_us":1989,"threads_started":1}
I20251027 14:54:33.669545 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.651215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41877 (local address 127.0.0.1:48252)
1027 14:54:33.652086 (+ 871us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.652135 (+ 49us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:33.653914 (+ 1779us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:33.654810 (+ 896us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:33.654818 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:33.655580 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:33.656076 (+ 496us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:33.656095 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:33.657589 (+ 1494us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:33.657594 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:33.658286 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:33.658296 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:33.658565 (+ 269us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:33.659398 (+ 833us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:33.659423 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:33.661977 (+ 2554us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:33.665738 (+ 3761us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:33.665748 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:33.665759 (+ 11us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:33.666145 (+ 386us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:33.666510 (+ 365us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:33.666514 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:33.666517 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:33.666626 (+ 109us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:33.667457 (+ 831us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:33.667463 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:33.667683 (+ 220us) client_negotiation.cc:769] Sending connection context
1027 14:54:33.667939 (+ 256us) client_negotiation.cc:240] Negotiation successful
1027 14:54:33.668583 (+ 644us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":619,"mutex_wait_us":138,"thread_start_us":474,"threads_started":1}
I20251027 14:54:33.672559 14700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:54:33.673594 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.673140 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:58990)
1027 14:54:33.673304 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.673427 (+ 123us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20251027 14:54:33.673838 14650 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
I20251027 14:54:33.675309 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.674946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:58992)
1027 14:54:33.675092 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.675169 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251027 14:54:33.675513 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38393: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.61@KRBTEST.COM" created.
I20251027 14:54:33.724843 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } attempt: 1
I20251027 14:54:33.725922 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.725452 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:58994)
1027 14:54:33.725663 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.725741 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20251027 14:54:33.726975 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.726560 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59006)
1027 14:54:33.726768 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.726845 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":105}
W20251027 14:54:33.727198 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38393: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:33.729293 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38393
--webserver_interface=127.0.104.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41877,127.0.104.61:38393,127.0.104.60:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:33.795432 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } attempt: 2
I20251027 14:54:33.796648 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.796185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59014)
1027 14:54:33.796412 (+ 227us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.796501 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
I20251027 14:54:33.797653 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.797305 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59030)
1027 14:54:33.797443 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.797505 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251027 14:54:33.797839 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38393: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
I20251027 14:54:33.889040 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } attempt: 3
I20251027 14:54:33.890040 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.889663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59046)
1027 14:54:33.889831 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.889912 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20251027 14:54:33.890926 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:33.890624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59050)
1027 14:54:33.890737 (+ 113us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:33.890799 (+ 62us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251027 14:54:33.891093 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38393: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
I20251027 14:54:34.034317 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } attempt: 4
I20251027 14:54:34.035535 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.035004 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59052)
1027 14:54:34.035289 (+ 285us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.035377 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":164}
I20251027 14:54:34.036530 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.036141 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59068)
1027 14:54:34.036290 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.036358 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
W20251027 14:54:34.036706 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:38393: Network error: Client connection negotiation failed: client connection to 127.0.104.61:38393: connect: Connection refused (error 111)
W20251027 14:54:34.047133 14709 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:34.047444 14709 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:34.047576 14709 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:34.055650 14709 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:34.055763 14709 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:34.055852 14709 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:34.055913 14709 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:34.055994 14709 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:34.066547 14709 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41877,127.0.104.61:38393,127.0.104.60:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:38393
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:34.068168 14709 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:34.069510 14709 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:34.077450 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:34.077428 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:34.078063 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:34.079042 14709 server_base.cc:1047] running on GCE node
I20251027 14:54:34.079823 14709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:34.081296 14709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:34.082512 14709 hybrid_clock.cc:648] HybridClock initialized: now 1761576874082441 us; error 44 us; skew 500 ppm
I20251027 14:54:34.082845 14709 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:34.085999 14709 init.cc:377] Logged in from keytab as kudu/127.0.104.61@KRBTEST.COM (short username kudu)
I20251027 14:54:34.087515 14709 webserver.cc:492] Webserver started at http://127.0.104.61:33597/ using document root <none> and password file <none>
I20251027 14:54:34.088265 14709 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:34.088346 14709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:34.088682 14709 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:34.091291 14709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data/instance:
uuid: "23d4c37784c947bca92e71cc7cef9862"
format_stamp: "Formatted at 2025-10-27 14:54:34 on dist-test-slave-hxxt"
server_key: "27f4f296c1cbb7e0d132afc1123afa74"
server_key_iv: "92e5d25a48bf1e22aefe29e3d363db34"
server_key_version: "encryptionkey@0"
I20251027 14:54:34.092052 14709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal/instance:
uuid: "23d4c37784c947bca92e71cc7cef9862"
format_stamp: "Formatted at 2025-10-27 14:54:34 on dist-test-slave-hxxt"
server_key: "27f4f296c1cbb7e0d132afc1123afa74"
server_key_iv: "92e5d25a48bf1e22aefe29e3d363db34"
server_key_version: "encryptionkey@0"
I20251027 14:54:34.097460 14709 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251027 14:54:34.100940 14724 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:34.102257 14709 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251027 14:54:34.102414 14709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal
uuid: "23d4c37784c947bca92e71cc7cef9862"
format_stamp: "Formatted at 2025-10-27 14:54:34 on dist-test-slave-hxxt"
server_key: "27f4f296c1cbb7e0d132afc1123afa74"
server_key_iv: "92e5d25a48bf1e22aefe29e3d363db34"
server_key_version: "encryptionkey@0"
I20251027 14:54:34.102632 14709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:34.120100 14709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:34.124414 14709 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:34.124718 14709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:34.143096 14709 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:38393
I20251027 14:54:34.143115 14775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:38393 every 8 connection(s)
I20251027 14:54:34.144601 14709 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
I20251027 14:54:34.148442 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14709
I20251027 14:54:34.148670 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-1/wal/instance
I20251027 14:54:34.148909 14776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:34.149284 416 external_mini_cluster.cc:1468] Setting key 0dded8bcebe19dcafb1885eb3810d05e
I20251027 14:54:34.157850 14776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.104.60@KRBTEST.COM; defaulting to no policy
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.60@KRBTEST.COM" created.
I20251027 14:54:34.176915 14704 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.160571 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48256 (local address 127.0.104.62:41877)
1027 14:54:34.160742 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.160747 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.162281 (+ 1534us) server_negotiation.cc:408] Connection header received
1027 14:54:34.163797 (+ 1516us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.163804 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.163896 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.163986 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.165856 (+ 1870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.166703 (+ 847us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.167775 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.168256 (+ 481us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.171205 (+ 2949us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.171241 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.171245 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.171279 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.173865 (+ 2586us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.174592 (+ 727us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.174597 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.174599 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.174675 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.175012 (+ 337us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.175016 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.175019 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.175270 (+ 251us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.175403 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.176243 (+ 840us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.176590 (+ 347us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:54:34.177874 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.161219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41877 (local address 127.0.0.1:48256)
1027 14:54:34.162095 (+ 876us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.162137 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.163463 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.164262 (+ 799us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.164278 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.164924 (+ 646us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.165658 (+ 734us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.165678 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.166870 (+ 1192us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.166875 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.167633 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.167644 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.167929 (+ 285us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.168786 (+ 857us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.168814 (+ 28us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.170974 (+ 2160us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.174023 (+ 3049us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.174033 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.174053 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.174460 (+ 407us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.174773 (+ 313us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.174777 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.174780 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.174900 (+ 120us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.175402 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.175411 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.175803 (+ 392us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.176103 (+ 300us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.176659 (+ 556us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":645,"thread_start_us":405,"threads_started":1}
I20251027 14:54:34.179849 14776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:54:34.194777 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.180689 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59074)
1027 14:54:34.180876 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.180894 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.181005 (+ 111us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.182065 (+ 1060us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.182071 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.182497 (+ 426us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.182919 (+ 422us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.182932 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.184525 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.184530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.185503 (+ 973us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.185516 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.185655 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.186405 (+ 750us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.186427 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.188727 (+ 2300us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.192247 (+ 3520us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.192253 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.192258 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.192680 (+ 422us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.193019 (+ 339us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.193034 (+ 15us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.193037 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.193109 (+ 72us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.193674 (+ 565us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.193680 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.193827 (+ 147us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.193942 (+ 115us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.194339 (+ 397us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":58}
I20251027 14:54:34.194856 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.180941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59074 (local address 127.0.104.61:38393)
1027 14:54:34.181603 (+ 662us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.181609 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.181636 (+ 27us) server_negotiation.cc:408] Connection header received
1027 14:54:34.181682 (+ 46us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.181688 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.181885 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.182029 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.183110 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.184350 (+ 1240us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.185636 (+ 1286us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.186072 (+ 436us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.188877 (+ 2805us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.188941 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.188957 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.188985 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.192106 (+ 3121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.192809 (+ 703us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.192817 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.192819 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.192887 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.193200 (+ 313us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.193205 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.193208 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.193523 (+ 315us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.193707 (+ 184us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.194075 (+ 368us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.194480 (+ 405us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":76,"server-negotiator.queue_time_us":507,"thread_start_us":354,"threads_started":1}
I20251027 14:54:34.197592 14776 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:34.198647 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.198257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51696)
1027 14:54:34.198412 (+ 155us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.198513 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":56}
W20251027 14:54:34.198930 14727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.200438 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.200126 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51698)
1027 14:54:34.200245 (+ 119us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.200305 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251027 14:54:34.200632 14776 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.104.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.60@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:34.240815 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42519
--webserver_interface=127.0.104.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:46075
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:41877,127.0.104.61:38393,127.0.104.60:42519
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:34.249935 14776 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 1
I20251027 14:54:34.250973 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.250515 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51706)
1027 14:54:34.250721 (+ 206us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.250811 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":86}
I20251027 14:54:34.251942 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.251621 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51718)
1027 14:54:34.251748 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.251838 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:54:34.252140 14776 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.320358 14776 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 2
I20251027 14:54:34.321484 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.321053 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51722)
1027 14:54:34.321242 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.321328 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20251027 14:54:34.322372 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.322078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51724)
1027 14:54:34.322198 (+ 120us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.322259 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:54:34.322595 14776 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.335956 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } attempt: 5
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:54:34.348683 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.336723 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59082 (local address 127.0.104.61:38393)
1027 14:54:34.336878 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.336883 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.336901 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:34.337096 (+ 195us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.337101 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.337163 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.337241 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.338415 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.339206 (+ 791us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.340555 (+ 1349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.340976 (+ 421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.343816 (+ 2840us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.343851 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.343855 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.343887 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.346357 (+ 2470us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.347120 (+ 763us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.347123 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.347126 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.347202 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.347483 (+ 281us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.347486 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.347488 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.347714 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.347816 (+ 102us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.348222 (+ 406us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.348443 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:54:34.348860 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.336588 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59082)
1027 14:54:34.336797 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.336820 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.336970 (+ 150us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.337302 (+ 332us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.337307 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.337785 (+ 478us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.338256 (+ 471us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.338271 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.339336 (+ 1065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.339342 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.340416 (+ 1074us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.340431 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.340576 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.341280 (+ 704us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.341304 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.343652 (+ 2348us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.346489 (+ 2837us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.346495 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.346499 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.346982 (+ 483us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.347297 (+ 315us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.347302 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.347305 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.347401 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.347813 (+ 412us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.347818 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.347979 (+ 161us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.348130 (+ 151us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.348499 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":57}
I20251027 14:54:34.349866 14700 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:54:34.350843 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.350415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51730)
1027 14:54:34.350590 (+ 175us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.350679 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20251027 14:54:34.351745 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.351409 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51746)
1027 14:54:34.351550 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.351626 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
W20251027 14:54:34.351955 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.403198 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 1
I20251027 14:54:34.404614 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.403918 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51762)
1027 14:54:34.404233 (+ 315us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.404368 (+ 135us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":93}
I20251027 14:54:34.406327 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.405844 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51764)
1027 14:54:34.406035 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.406177 (+ 142us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251027 14:54:34.406590 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.413801 14776 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 3
I20251027 14:54:34.414736 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.414327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51780)
1027 14:54:34.414497 (+ 170us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.414591 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20251027 14:54:34.415823 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.415393 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51782)
1027 14:54:34.415639 (+ 246us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.415703 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":169}
W20251027 14:54:34.416018 14776 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.474865 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 2
I20251027 14:54:34.476064 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.475557 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51798)
1027 14:54:34.475796 (+ 239us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.475900 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20251027 14:54:34.477049 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.476688 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51808)
1027 14:54:34.476852 (+ 164us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.476937 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
W20251027 14:54:34.477246 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.559196 14776 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 4
I20251027 14:54:34.560281 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.559880 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51810)
1027 14:54:34.560054 (+ 174us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.560135 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":59}
I20251027 14:54:34.561218 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.560917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51816)
1027 14:54:34.561039 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.561114 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:54:34.561414 14776 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
W20251027 14:54:34.567634 14785 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:34.567948 14785 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:34.568077 14785 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:34.576221 14785 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:34.576328 14785 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:34.576434 14785 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:34.576503 14785 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:34.576586 14785 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:34.583441 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 3
I20251027 14:54:34.584549 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.584065 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51820)
1027 14:54:34.584264 (+ 199us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.584357 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":72}
I20251027 14:54:34.585457 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.585113 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51826)
1027 14:54:34.585285 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.585346 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
W20251027 14:54:34.585651 14700 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42519: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42519: connect: Connection refused (error 111)
I20251027 14:54:34.587706 14785 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46075
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:41877,127.0.104.61:38393,127.0.104.60:42519
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42519
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:34.589399 14785 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:34.590782 14785 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:34.598727 14791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:34.598783 14790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:34.599516 14793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:34.600049 14785 server_base.cc:1047] running on GCE node
I20251027 14:54:34.600917 14785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:34.602326 14785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:34.603535 14785 hybrid_clock.cc:648] HybridClock initialized: now 1761576874603466 us; error 43 us; skew 500 ppm
I20251027 14:54:34.603864 14785 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:34.607087 14785 init.cc:377] Logged in from keytab as kudu/127.0.104.60@KRBTEST.COM (short username kudu)
I20251027 14:54:34.608839 14785 webserver.cc:492] Webserver started at http://127.0.104.60:36385/ using document root <none> and password file <none>
I20251027 14:54:34.609592 14785 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:34.609673 14785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:34.610031 14785 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:34.612776 14785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data/instance:
uuid: "a04be9bb5a6147f4a1e542e419c40dec"
format_stamp: "Formatted at 2025-10-27 14:54:34 on dist-test-slave-hxxt"
server_key: "0730bbd24212843be8c2f3869924c411"
server_key_iv: "2457b8196a84f4196e0119d0f9a31c36"
server_key_version: "encryptionkey@0"
I20251027 14:54:34.613544 14785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal/instance:
uuid: "a04be9bb5a6147f4a1e542e419c40dec"
format_stamp: "Formatted at 2025-10-27 14:54:34 on dist-test-slave-hxxt"
server_key: "0730bbd24212843be8c2f3869924c411"
server_key_iv: "2457b8196a84f4196e0119d0f9a31c36"
server_key_version: "encryptionkey@0"
I20251027 14:54:34.619195 14785 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:54:34.622891 14800 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:34.624312 14785 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:54:34.624557 14785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal
uuid: "a04be9bb5a6147f4a1e542e419c40dec"
format_stamp: "Formatted at 2025-10-27 14:54:34 on dist-test-slave-hxxt"
server_key: "0730bbd24212843be8c2f3869924c411"
server_key_iv: "2457b8196a84f4196e0119d0f9a31c36"
server_key_version: "encryptionkey@0"
I20251027 14:54:34.624785 14785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:34.651572 14785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:34.655550 14785 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:34.655809 14785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:34.674502 14785 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:42519
I20251027 14:54:34.674522 14851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:42519 every 8 connection(s)
I20251027 14:54:34.675992 14785 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
I20251027 14:54:34.680106 14852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:34.681921 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14785
I20251027 14:54:34.682132 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACert.1761576633729408-416-0/minicluster-data/master-2/wal/instance
I20251027 14:54:34.682621 416 external_mini_cluster.cc:1468] Setting key 2d1a91f86838ae11c2e8d9acb30eee3b
I20251027 14:54:34.687809 14852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:34.704654 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.684587 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51832 (local address 127.0.104.60:42519)
1027 14:54:34.685352 (+ 765us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.685366 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.685407 (+ 41us) server_negotiation.cc:408] Connection header received
1027 14:54:34.686663 (+ 1256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.686686 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.687119 (+ 433us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.687500 (+ 381us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.689618 (+ 2118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.691264 (+ 1646us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.692767 (+ 1503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.693305 (+ 538us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.696901 (+ 3596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.696978 (+ 77us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.696990 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.697020 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.700404 (+ 3384us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.701035 (+ 631us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.701044 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.701051 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.701141 (+ 90us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.701450 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.701456 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.701459 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.702087 (+ 628us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.702347 (+ 260us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.702787 (+ 440us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.703218 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":375,"threads_started":1}
I20251027 14:54:34.704872 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.690541 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.104.62:41877)
1027 14:54:34.691004 (+ 463us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.691011 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.691533 (+ 522us) server_negotiation.cc:408] Connection header received
1027 14:54:34.691726 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.691732 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.691825 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.691935 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.693278 (+ 1343us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.694254 (+ 976us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.695571 (+ 1317us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.696106 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.698690 (+ 2584us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.698752 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.698758 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.698798 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.701537 (+ 2739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.702993 (+ 1456us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.702997 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.703005 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.703084 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.703468 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.703471 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.703473 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.703712 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.703816 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.704315 (+ 499us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.704572 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":236,"threads_started":1}
I20251027 14:54:34.705000 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.690530 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:41877 (local address 127.0.0.1:48268)
1027 14:54:34.691383 (+ 853us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.691424 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.691597 (+ 173us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.691990 (+ 393us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.691995 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.692623 (+ 628us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.693080 (+ 457us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.693092 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.694400 (+ 1308us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.694406 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.695428 (+ 1022us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.695440 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.695609 (+ 169us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.696183 (+ 574us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.696207 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.698487 (+ 2280us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.701649 (+ 3162us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.701659 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.701666 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.702110 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.703218 (+ 1108us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.703223 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.703226 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.703373 (+ 147us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.703824 (+ 451us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.703832 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.704004 (+ 172us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.704254 (+ 250us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.704609 (+ 355us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":652,"thread_start_us":469,"threads_started":1}
I20251027 14:54:34.706489 14852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:34.718449 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.707161 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:38393 (local address 127.0.0.1:59096)
1027 14:54:34.707327 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.707342 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.707444 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.707791 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.707795 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.708089 (+ 294us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.708547 (+ 458us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.708559 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.709649 (+ 1090us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.709655 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.710347 (+ 692us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.710357 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.710486 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.711149 (+ 663us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.711166 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.713245 (+ 2079us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.716307 (+ 3062us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.716313 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.716317 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.716673 (+ 356us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.716979 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.716984 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.716988 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.717080 (+ 92us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.717660 (+ 580us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.717664 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.717840 (+ 176us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.717990 (+ 150us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.718227 (+ 237us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":67}
I20251027 14:54:34.718881 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.707312 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59096 (local address 127.0.104.61:38393)
1027 14:54:34.707512 (+ 200us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.707518 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.707534 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:54:34.707588 (+ 54us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.707592 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.707664 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.707772 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.708719 (+ 947us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.709519 (+ 800us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.710489 (+ 970us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.710912 (+ 423us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.713422 (+ 2510us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.713461 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.713465 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.713496 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.716175 (+ 2679us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.716789 (+ 614us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.716796 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.716799 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.716878 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.717181 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.717186 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.717189 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.717501 (+ 312us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.717659 (+ 158us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.718107 (+ 448us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.718358 (+ 251us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":87}
I20251027 14:54:34.719772 14852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:54:34.721848 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.708446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48270 (local address 127.0.104.62:41877)
1027 14:54:34.708602 (+ 156us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.708608 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.708624 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:54:34.708786 (+ 162us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.708792 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.708851 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.708922 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.710186 (+ 1264us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.711100 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.712248 (+ 1148us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.712760 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.716336 (+ 3576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.716362 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.716366 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.716424 (+ 58us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.719521 (+ 3097us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.720170 (+ 649us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.720174 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.720177 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.720242 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.720585 (+ 343us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.720589 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.720591 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.720802 (+ 211us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.721036 (+ 234us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.721339 (+ 303us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.721578 (+ 239us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:54:34.731761 14857 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.720504 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51844)
1027 14:54:34.720713 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.720729 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.720828 (+ 99us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.721080 (+ 252us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.721085 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.721408 (+ 323us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.721747 (+ 339us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.721760 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.723013 (+ 1253us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.723019 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.723751 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.723764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.723893 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.724589 (+ 696us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.724606 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.726573 (+ 1967us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.729822 (+ 3249us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.729828 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.729832 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.730127 (+ 295us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.730381 (+ 254us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.730386 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.730389 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.730452 (+ 63us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.730851 (+ 399us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.730856 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.730985 (+ 129us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.731111 (+ 126us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.731386 (+ 275us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":137}
I20251027 14:54:34.731807 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.720521 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51844 (local address 127.0.104.60:42519)
1027 14:54:34.720667 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.720672 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.720787 (+ 115us) server_negotiation.cc:408] Connection header received
1027 14:54:34.720930 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.720934 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.720983 (+ 49us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.721052 (+ 69us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.721898 (+ 846us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.722902 (+ 1004us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.723883 (+ 981us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.724357 (+ 474us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.726731 (+ 2374us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.726777 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.726781 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.726806 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.729705 (+ 2899us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.730229 (+ 524us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.730235 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.730238 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.730297 (+ 59us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.730534 (+ 237us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.730539 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.730542 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.730751 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.730870 (+ 119us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.731203 (+ 333us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.731522 (+ 319us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":109,"server-negotiator.queue_time_us":56}
I20251027 14:54:34.737398 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.723556 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59098 (local address 127.0.104.61:38393)
1027 14:54:34.723718 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.723724 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.723750 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:54:34.723864 (+ 114us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.723869 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.723940 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.724029 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.725824 (+ 1795us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.726940 (+ 1116us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.728206 (+ 1266us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.728702 (+ 496us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.731751 (+ 3049us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.731777 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.731780 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.731809 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.735076 (+ 3267us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.735740 (+ 664us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.735746 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.735750 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.735835 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.736083 (+ 248us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.736087 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.736090 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.736359 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.736691 (+ 332us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.736848 (+ 157us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.737078 (+ 230us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:54:34.737994 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: Bootstrap starting.
I20251027 14:54:34.741809 14852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:34.742206 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:54:34.743070 14852 log.cc:826] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:34.746356 14852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: No bootstrap required, opened a new log
I20251027 14:54:34.753111 14852 raft_consensus.cc:359] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.753402 14852 raft_consensus.cc:385] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:34.753494 14852 raft_consensus.cc:740] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a04be9bb5a6147f4a1e542e419c40dec, State: Initialized, Role: FOLLOWER
I20251027 14:54:34.754076 14852 consensus_queue.cc:260] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.754978 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:34.755249 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:54:34.755874 14852 sys_catalog.cc:565] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:34.762828 14700 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 4
W20251027 14:54:34.765245 14871 catalog_manager.cc:1568] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 14:54:34.765496 14871 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 14:54:34.763397 14785 master_runner.cc:186] Error getting master registration for 127.0.104.62:41877: OK, instance_id { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" instance_seqno: 1761576873628421 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 14:54:34.769567 14785 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
I20251027 14:54:34.780654 14703 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.763368 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51860)
1027 14:54:34.763698 (+ 330us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.763717 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.763840 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.766309 (+ 2469us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.766315 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.766809 (+ 494us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.767251 (+ 442us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.767265 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.769120 (+ 1855us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.769127 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.770225 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.770242 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.770397 (+ 155us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.771161 (+ 764us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.771188 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.773907 (+ 2719us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.778051 (+ 4144us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.778057 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.778062 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.778459 (+ 397us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.778820 (+ 361us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.778825 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.778829 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.778937 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.779622 (+ 685us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.779628 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.779784 (+ 156us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.779940 (+ 156us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.780250 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":217}
I20251027 14:54:34.781067 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.764026 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51860 (local address 127.0.104.60:42519)
1027 14:54:34.765799 (+ 1773us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.765805 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.765829 (+ 24us) server_negotiation.cc:408] Connection header received
1027 14:54:34.765899 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.765905 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.765982 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.766092 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.767442 (+ 1350us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.768920 (+ 1478us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.770394 (+ 1474us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.770855 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.774078 (+ 3223us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.774121 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.774127 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.774165 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.777861 (+ 3696us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.778587 (+ 726us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.778593 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.778596 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.778705 (+ 109us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.779053 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.779059 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.779062 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.779494 (+ 432us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.779638 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.780050 (+ 412us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.780311 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1628}
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20251027 14:54:34.787398 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: Bootstrap starting.
I20251027 14:54:34.791702 14700 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:34.793260 14700 log.cc:826] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:34.807394 14700 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: No bootstrap required, opened a new log
I20251027 14:54:34.815546 14700 raft_consensus.cc:359] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.815954 14700 raft_consensus.cc:385] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:34.816092 14700 raft_consensus.cc:740] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cee2ec5a9da34357be142cb158e21dc2, State: Initialized, Role: FOLLOWER
I20251027 14:54:34.817023 14700 consensus_queue.cc:260] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:54:34.824126 14882 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:34.824465 14882 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: This master's current role is: FOLLOWER
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
I20251027 14:54:34.826530 14700 sys_catalog.cc:565] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:34.831115 14856 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.791382 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48284 (local address 127.0.104.62:41877)
1027 14:54:34.791878 (+ 496us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.791886 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.803456 (+ 11570us) server_negotiation.cc:408] Connection header received
1027 14:54:34.803955 (+ 499us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.803963 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.804088 (+ 125us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.804166 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.806822 (+ 2656us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.808274 (+ 1452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.810526 (+ 2252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.811047 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.821541 (+ 10494us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.821582 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.821588 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.821624 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.826414 (+ 4790us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.828184 (+ 1770us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.828191 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.828194 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.828291 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.828782 (+ 491us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.828787 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.828790 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.829111 (+ 321us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.829364 (+ 253us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.830450 (+ 1086us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.830748 (+ 298us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371}
I20251027 14:54:34.835484 14781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.792041 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59106 (local address 127.0.104.61:38393)
1027 14:54:34.792222 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.792229 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.794253 (+ 2024us) server_negotiation.cc:408] Connection header received
1027 14:54:34.794556 (+ 303us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.794563 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.794648 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.794722 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.800369 (+ 5647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.801680 (+ 1311us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.814024 (+ 12344us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.814577 (+ 553us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.821346 (+ 6769us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.821395 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.821401 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.821460 (+ 59us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.829905 (+ 8445us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.832638 (+ 2733us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.832645 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.832648 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.832782 (+ 134us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.833232 (+ 450us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.833238 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.833242 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.833586 (+ 344us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.833871 (+ 285us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.834843 (+ 972us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.835105 (+ 262us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251027 14:54:34.836094 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.792845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51870 (local address 127.0.104.60:42519)
1027 14:54:34.792990 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.792995 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.812305 (+ 19310us) server_negotiation.cc:408] Connection header received
1027 14:54:34.812659 (+ 354us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.812666 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.812754 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.812831 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.816123 (+ 3292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.817623 (+ 1500us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.819538 (+ 1915us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.820007 (+ 469us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.826546 (+ 6539us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.826595 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.826601 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.826644 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.831730 (+ 5086us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.833572 (+ 1842us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.833578 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.833581 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.833713 (+ 132us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.834141 (+ 428us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.834147 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.834150 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.834439 (+ 289us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.834680 (+ 241us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.835184 (+ 504us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.835456 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":41}
W20251027 14:54:34.851200 14893 catalog_manager.cc:1568] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 14:54:34.851418 14893 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 14:54:34.846191 14633 master_runner.cc:186] Error getting master registration for 127.0.104.61:38393: OK, instance_id { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" instance_seqno: 1761576874138065 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 14:54:34.852617 14633 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251027 14:54:34.860594 14776 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } attempt: 5
W20251027 14:54:34.867259 14654 tablet.cc:2378] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: Can't schedule compaction. Clean time has not been advanced past its initial value.
Oct 27 14:54:34 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
I20251027 14:54:34.876958 14779 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.861180 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42519 (local address 127.0.0.1:51878)
1027 14:54:34.861627 (+ 447us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:34.861645 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:34.861772 (+ 127us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:34.862327 (+ 555us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:34.862334 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:34.862855 (+ 521us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:34.863325 (+ 470us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.863341 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.865071 (+ 1730us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.865077 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:34.866144 (+ 1067us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:34.866161 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.866306 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.867053 (+ 747us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:34.867080 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:34.869868 (+ 2788us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:34.874259 (+ 4391us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.874266 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.874271 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.874749 (+ 478us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.875104 (+ 355us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:34.875110 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:34.875113 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:34.875217 (+ 104us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:34.875913 (+ 696us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:34.875918 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:34.876063 (+ 145us) client_negotiation.cc:769] Sending connection context
1027 14:54:34.876212 (+ 149us) client_negotiation.cc:240] Negotiation successful
1027 14:54:34.876584 (+ 372us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":319}
I20251027 14:54:34.876987 14855 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:34.861265 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51878 (local address 127.0.104.60:42519)
1027 14:54:34.861471 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:34.861478 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:34.861880 (+ 402us) server_negotiation.cc:408] Connection header received
1027 14:54:34.861945 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:34.861951 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:34.862025 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:34.862116 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:34.863503 (+ 1387us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.864892 (+ 1389us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:34.866297 (+ 1405us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:34.866882 (+ 585us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:34.870028 (+ 3146us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:34.870089 (+ 61us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:34.870095 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:34.870132 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:34.874095 (+ 3963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.874888 (+ 793us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.874894 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.874898 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.874983 (+ 85us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:34.875332 (+ 349us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:34.875337 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:34.875341 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:34.875693 (+ 352us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:34.875850 (+ 157us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:34.876358 (+ 508us) server_negotiation.cc:300] Negotiation successful
1027 14:54:34.876632 (+ 274us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251027 14:54:34.883699 14776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: Bootstrap starting.
I20251027 14:54:34.888072 14776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:34.889704 14776 log.cc:826] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:34.894558 14776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: No bootstrap required, opened a new log
I20251027 14:54:34.902424 14776 raft_consensus.cc:359] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.902822 14776 raft_consensus.cc:385] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:34.902961 14776 raft_consensus.cc:740] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23d4c37784c947bca92e71cc7cef9862, State: Initialized, Role: FOLLOWER
I20251027 14:54:34.903733 14776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.904852 14896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:34.905200 14896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:54:34.906262 14776 sys_catalog.cc:565] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: configured and running, proceeding with master startup.
W20251027 14:54:34.907862 14806 tablet.cc:2378] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:54:34.930344 14709 master_runner.cc:263] No leader master found from master 23d4c37784c947bca92e71cc7cef9862
I20251027 14:54:34.930500 14709 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251027 14:54:34.938915 14907 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 14:54:34.939062 14907 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 14:54:34.945313 14882 raft_consensus.cc:493] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:34.945596 14882 raft_consensus.cc:515] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.947314 14882 leader_election.cc:290] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 23d4c37784c947bca92e71cc7cef9862 (127.0.104.61:38393), a04be9bb5a6147f4a1e542e419c40dec (127.0.104.60:42519)
I20251027 14:54:34.949827 14827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cee2ec5a9da34357be142cb158e21dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04be9bb5a6147f4a1e542e419c40dec" is_pre_election: true
I20251027 14:54:34.950318 14827 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee2ec5a9da34357be142cb158e21dc2 in term 0.
I20251027 14:54:34.950932 14651 leader_election.cc:304] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: a04be9bb5a6147f4a1e542e419c40dec, cee2ec5a9da34357be142cb158e21dc2; no voters:
I20251027 14:54:34.951280 14751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cee2ec5a9da34357be142cb158e21dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d4c37784c947bca92e71cc7cef9862" is_pre_election: true
I20251027 14:54:34.951431 14882 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:54:34.951592 14882 raft_consensus.cc:493] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:54:34.951691 14882 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:34.951735 14751 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cee2ec5a9da34357be142cb158e21dc2 in term 0.
I20251027 14:54:34.955443 14882 raft_consensus.cc:515] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.956521 14751 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cee2ec5a9da34357be142cb158e21dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "23d4c37784c947bca92e71cc7cef9862"
I20251027 14:54:34.956704 14751 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:34.960739 14751 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee2ec5a9da34357be142cb158e21dc2 in term 1.
I20251027 14:54:34.961297 14650 leader_election.cc:304] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 23d4c37784c947bca92e71cc7cef9862, cee2ec5a9da34357be142cb158e21dc2; no voters:
I20251027 14:54:34.964816 14827 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "cee2ec5a9da34357be142cb158e21dc2" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a04be9bb5a6147f4a1e542e419c40dec"
I20251027 14:54:34.965034 14827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:34.970933 14827 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cee2ec5a9da34357be142cb158e21dc2 in term 1.
I20251027 14:54:34.973505 14882 leader_election.cc:290] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [CANDIDATE]: Term 1 election: Requested vote from peers 23d4c37784c947bca92e71cc7cef9862 (127.0.104.61:38393), a04be9bb5a6147f4a1e542e419c40dec (127.0.104.60:42519)
I20251027 14:54:34.974017 14882 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:34.974603 14882 raft_consensus.cc:697] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [term 1 LEADER]: Becoming Leader. State: Replica: cee2ec5a9da34357be142cb158e21dc2, State: Running, Role: LEADER
I20251027 14:54:34.975397 14882 consensus_queue.cc:237] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } }
I20251027 14:54:34.994572 14909 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee2ec5a9da34357be142cb158e21dc2. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:34.995052 14909 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:35.009866 14911 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:35.012276 14911 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:35.039594 14827 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [term 1 FOLLOWER]: Refusing update from remote peer cee2ec5a9da34357be142cb158e21dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:35.040700 14909 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:35.048074 14751 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [term 1 FOLLOWER]: Refusing update from remote peer cee2ec5a9da34357be142cb158e21dc2: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:35.049082 14909 consensus_queue.cc:1048] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:35.074195 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee2ec5a9da34357be142cb158e21dc2. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:35.074473 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:54:35.079440 14912 mvcc.cc:204] Tried to move back new op lower bound from 7215418880122920960 to 7215418879913005056. Current Snapshot: MvccSnapshot[applied={T|T < 7215418880122920960}]
I20251027 14:54:35.084901 14882 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:35.085230 14882 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:35.088349 14896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: SysCatalogTable state changed. Reason: New leader cee2ec5a9da34357be142cb158e21dc2. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:35.088639 14896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:54:35.088984 14911 catalog_manager.cc:1357] Generated new cluster ID: 5179bd0eed594e8f9b65a875b48d2b77
I20251027 14:54:35.089088 14911 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:54:35.093024 14860 sys_catalog.cc:455] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:35.093223 14860 sys_catalog.cc:458] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:54:35.116051 14909 sys_catalog.cc:455] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:35.117697 14896 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "cee2ec5a9da34357be142cb158e21dc2" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "cee2ec5a9da34357be142cb158e21dc2" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 41877 } } peers { permanent_uuid: "23d4c37784c947bca92e71cc7cef9862" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 38393 } } peers { permanent_uuid: "a04be9bb5a6147f4a1e542e419c40dec" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42519 } } }
I20251027 14:54:35.117905 14896 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:54:35.118391 14909 sys_catalog.cc:458] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2 [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:35.156169 14911 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:54:35.158048 14911 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:54:35.182951 14911 catalog_manager.cc:6022] T 00000000000000000000000000000000 P cee2ec5a9da34357be142cb158e21dc2: Generated new TSK 0
I20251027 14:54:35.184224 14911 catalog_manager.cc:1524] Initializing in-progress tserver states...
Oct 27 14:54:35 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.104.62@KRBTEST.COM
Oct 27 14:54:35 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.104.61@KRBTEST.COM
Oct 27 14:54:35 dist-test-slave-hxxt krb5kdc[14617](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576874, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.104.60@KRBTEST.COM
I20251027 14:54:35.767578 14871 catalog_manager.cc:1269] Loaded cluster ID: 5179bd0eed594e8f9b65a875b48d2b77
I20251027 14:54:35.767683 14871 catalog_manager.cc:1562] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: loading cluster ID for follower catalog manager: success
I20251027 14:54:35.770191 14871 catalog_manager.cc:1584] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: acquiring CA information for follower catalog manager: success
I20251027 14:54:35.771966 14871 catalog_manager.cc:1612] T 00000000000000000000000000000000 P a04be9bb5a6147f4a1e542e419c40dec: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 14:54:35.941038 14907 catalog_manager.cc:1269] Loaded cluster ID: 5179bd0eed594e8f9b65a875b48d2b77
I20251027 14:54:35.941144 14907 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: loading cluster ID for follower catalog manager: success
I20251027 14:54:35.943637 14907 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: acquiring CA information for follower catalog manager: success
I20251027 14:54:35.945475 14907 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 23d4c37784c947bca92e71cc7cef9862: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 14:54:35.960537 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14633
I20251027 14:54:35.961557 14695 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:36.070825 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14633
W20251027 14:54:36.087932 14801 connection.cc:537] server connection from 127.0.0.1:51860 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 14:54:36.088301 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14709
I20251027 14:54:36.089366 14771 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:36.195884 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14709
I20251027 14:54:36.212663 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14785
I20251027 14:54:36.213692 14847 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:36.320058 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14785
2025-10-27T14:54:36Z chronyd exiting
[ OK ] SecurityITest.IPKICACert (5255 ms)
[ RUN ] SecurityITest.IPKICACertWebServerEndPoints
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:54:36 dist-test-slave-hxxt krb5kdc[14931](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:54:36 dist-test-slave-hxxt krb5kdc[14931](info): set up 2 sockets
Oct 27 14:54:36 dist-test-slave-hxxt krb5kdc[14931](info): commencing operation
krb5kdc: starting...
W20251027 14:54:38.216802 416 mini_kdc.cc:121] Time spent starting KDC: real 1.856s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:54:38 dist-test-slave-hxxt krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:54:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:54:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:38.467365 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35649
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42311
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35649
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:38.791431 14947 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:38.791750 14947 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:38.791884 14947 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:38.800110 14947 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:38.800225 14947 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:38.800305 14947 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:38.800391 14947 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:38.800467 14947 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:38.810974 14947 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42311
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35649
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35649
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:38.812615 14947 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:38.813973 14947 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:38.822057 14952 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:38.822263 14955 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:38.822028 14953 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:38.823041 14947 server_base.cc:1047] running on GCE node
I20251027 14:54:38.823768 14947 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:38.825145 14947 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:38.826370 14947 hybrid_clock.cc:648] HybridClock initialized: now 1761576878826290 us; error 46 us; skew 500 ppm
I20251027 14:54:38.826700 14947 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:38 dist-test-slave-hxxt krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576878, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:38.829660 14947 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:38.831149 14947 webserver.cc:492] Webserver started at http://127.0.104.62:46867/ using document root <none> and password file <none>
I20251027 14:54:38.831854 14947 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:38.831941 14947 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:38.832284 14947 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:38.834797 14947 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "22ab14cf788e4c5b8d2db621d482ebed"
format_stamp: "Formatted at 2025-10-27 14:54:38 on dist-test-slave-hxxt"
server_key: "23a020d1ea57bbfec41022d6f65e63ec"
server_key_iv: "ecde685cc17958cb034485d734d7d834"
server_key_version: "encryptionkey@0"
I20251027 14:54:38.835537 14947 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "22ab14cf788e4c5b8d2db621d482ebed"
format_stamp: "Formatted at 2025-10-27 14:54:38 on dist-test-slave-hxxt"
server_key: "23a020d1ea57bbfec41022d6f65e63ec"
server_key_iv: "ecde685cc17958cb034485d734d7d834"
server_key_version: "encryptionkey@0"
I20251027 14:54:38.840757 14947 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251027 14:54:38.843977 14962 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:38.845392 14947 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:38.845557 14947 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "22ab14cf788e4c5b8d2db621d482ebed"
format_stamp: "Formatted at 2025-10-27 14:54:38 on dist-test-slave-hxxt"
server_key: "23a020d1ea57bbfec41022d6f65e63ec"
server_key_iv: "ecde685cc17958cb034485d734d7d834"
server_key_version: "encryptionkey@0"
I20251027 14:54:38.845749 14947 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:38.874891 14947 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:38.882969 14947 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:38.883239 14947 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:38.902048 14947 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35649
I20251027 14:54:38.902063 15013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35649 every 8 connection(s)
I20251027 14:54:38.903525 14947 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:38.907605 15014 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:38.908311 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 14947
I20251027 14:54:38.908560 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.IPKICACertWebServerEndPoints.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:54:38.909108 416 external_mini_cluster.cc:1468] Setting key 098a0afbc07d91d4ee3a08fcdc7449c6
I20251027 14:54:38.920643 15014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed: Bootstrap starting.
Oct 27 14:54:38 dist-test-slave-hxxt krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:38.923913 15014 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:38.924993 15014 log.cc:826] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:38.927783 15014 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed: No bootstrap required, opened a new log
I20251027 14:54:38.929837 15017 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:38.910868 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35966 (local address 127.0.104.62:35649)
1027 14:54:38.911662 (+ 794us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:38.911673 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:38.911716 (+ 43us) server_negotiation.cc:408] Connection header received
1027 14:54:38.912877 (+ 1161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:38.912900 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:38.913297 (+ 397us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:38.913676 (+ 379us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:38.915625 (+ 1949us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:38.917039 (+ 1414us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:38.918763 (+ 1724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:38.919288 (+ 525us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:38.923052 (+ 3764us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:38.923146 (+ 94us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:38.923162 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:38.923200 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:38.926181 (+ 2981us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:38.926831 (+ 650us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:38.926838 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:38.926845 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:38.926942 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:38.927269 (+ 327us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:38.927274 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:38.927278 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:38.927713 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:38.928066 (+ 353us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:38.928464 (+ 398us) server_negotiation.cc:300] Negotiation successful
1027 14:54:38.928859 (+ 395us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":611,"thread_start_us":394,"threads_started":1}
I20251027 14:54:38.933621 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:54:38.933408 15014 raft_consensus.cc:359] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ab14cf788e4c5b8d2db621d482ebed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35649 } }
I20251027 14:54:38.933760 15014 raft_consensus.cc:385] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:38.933854 15014 raft_consensus.cc:740] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22ab14cf788e4c5b8d2db621d482ebed, State: Initialized, Role: FOLLOWER
I20251027 14:54:38.934409 15014 consensus_queue.cc:260] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ab14cf788e4c5b8d2db621d482ebed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35649 } }
I20251027 14:54:38.934602 15014 raft_consensus.cc:399] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:54:38.934718 15014 raft_consensus.cc:493] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:54:38.934854 15014 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:38.937044 15014 raft_consensus.cc:515] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ab14cf788e4c5b8d2db621d482ebed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35649 } }
I20251027 14:54:38.937512 15014 leader_election.cc:304] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 22ab14cf788e4c5b8d2db621d482ebed; no voters:
I20251027 14:54:38.938139 15014 leader_election.cc:290] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:54:38.938251 15019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:38.940344 15014 sys_catalog.cc:565] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:38.940636 15019 raft_consensus.cc:697] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [term 1 LEADER]: Becoming Leader. State: Replica: 22ab14cf788e4c5b8d2db621d482ebed, State: Running, Role: LEADER
I20251027 14:54:38.941201 15019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ab14cf788e4c5b8d2db621d482ebed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35649 } }
I20251027 14:54:38.949788 15021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 22ab14cf788e4c5b8d2db621d482ebed. Latest consensus state: current_term: 1 leader_uuid: "22ab14cf788e4c5b8d2db621d482ebed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ab14cf788e4c5b8d2db621d482ebed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35649 } } }
I20251027 14:54:38.950043 15021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:38.950439 15020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "22ab14cf788e4c5b8d2db621d482ebed" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "22ab14cf788e4c5b8d2db621d482ebed" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35649 } } }
I20251027 14:54:38.950708 15020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:38.951035 15037 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:38.955725 15037 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:38.965530 15037 catalog_manager.cc:1357] Generated new cluster ID: a1da3701b03341dba2fb927608cbedcb
I20251027 14:54:38.965651 15037 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
Oct 27 14:54:38 dist-test-slave-hxxt krb5kdc[14931](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20251027 14:54:38.988798 15037 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:54:38.989993 15037 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:54:39.000280 15037 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 22ab14cf788e4c5b8d2db621d482ebed: Generated new TSK 0
I20251027 14:54:39.001165 15037 catalog_manager.cc:1524] Initializing in-progress tserver states...
Oct 27 14:54:39 dist-test-slave-hxxt krb5kdc[14931](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576878, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.104.62@KRBTEST.COM
I20251027 14:54:39.058565 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 14947
I20251027 14:54:39.059729 15009 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:39.155783 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 14947
2025-10-27T14:54:39Z chronyd exiting
[ OK ] SecurityITest.IPKICACertWebServerEndPoints (2829 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedIPKICert
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:54:39 dist-test-slave-hxxt krb5kdc[15045](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:54:39 dist-test-slave-hxxt krb5kdc[15045](info): set up 2 sockets
Oct 27 14:54:39 dist-test-slave-hxxt krb5kdc[15045](info): commencing operation
krb5kdc: starting...
W20251027 14:54:41.032677 416 mini_kdc.cc:121] Time spent starting KDC: real 1.842s user 0.000s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:54:41 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:54:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:54:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:41.287369 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39727
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:41.606349 15061 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:41.606662 15061 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:41.606804 15061 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:41.615042 15061 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:41.615149 15061 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:41.615271 15061 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:41.615407 15061 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:41.615489 15061 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:41.626358 15061 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:39727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:41.627969 15061 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:41.629352 15061 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:41.637178 15067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:41.637197 15066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:41.638595 15069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:41.639153 15061 server_base.cc:1047] running on GCE node
I20251027 14:54:41.639889 15061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:41.641342 15061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:41.642546 15061 hybrid_clock.cc:648] HybridClock initialized: now 1761576881642480 us; error 39 us; skew 500 ppm
I20251027 14:54:41.642861 15061 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:41 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576881, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:41.645915 15061 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:41.647532 15061 webserver.cc:492] Webserver started at http://127.0.104.62:36661/ using document root <none> and password file <none>
I20251027 14:54:41.648267 15061 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:41.648356 15061 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:41.648706 15061 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:41.651307 15061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "e625969dad8747ffb083fc47c71eb5bd"
format_stamp: "Formatted at 2025-10-27 14:54:41 on dist-test-slave-hxxt"
server_key: "01a4d8b63c49655046550a6011f8b748"
server_key_iv: "aab154fc1c62670a4107b7abdbd61c94"
server_key_version: "encryptionkey@0"
I20251027 14:54:41.652030 15061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "e625969dad8747ffb083fc47c71eb5bd"
format_stamp: "Formatted at 2025-10-27 14:54:41 on dist-test-slave-hxxt"
server_key: "01a4d8b63c49655046550a6011f8b748"
server_key_iv: "aab154fc1c62670a4107b7abdbd61c94"
server_key_version: "encryptionkey@0"
I20251027 14:54:41.657361 15061 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251027 14:54:41.660588 15076 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:41.661819 15061 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:54:41.661988 15061 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "e625969dad8747ffb083fc47c71eb5bd"
format_stamp: "Formatted at 2025-10-27 14:54:41 on dist-test-slave-hxxt"
server_key: "01a4d8b63c49655046550a6011f8b748"
server_key_iv: "aab154fc1c62670a4107b7abdbd61c94"
server_key_version: "encryptionkey@0"
I20251027 14:54:41.662179 15061 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:41.699569 15061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:41.703464 15061 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:41.703719 15061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:41.722622 15061 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39727
I20251027 14:54:41.722642 15127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39727 every 8 connection(s)
I20251027 14:54:41.724102 15061 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:41.727939 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15061
I20251027 14:54:41.728125 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:54:41.728632 15128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:41.728763 416 external_mini_cluster.cc:1468] Setting key 2b8ef29c16634f7a6c7f204a3bd29d62
Oct 27 14:54:41 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:41.742410 15128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap starting.
I20251027 14:54:41.746667 15128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:41.747890 15128 log.cc:826] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:41.751005 15131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:41.730766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42696 (local address 127.0.104.62:39727)
1027 14:54:41.731558 (+ 792us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:41.731573 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:41.731633 (+ 60us) server_negotiation.cc:408] Connection header received
1027 14:54:41.733137 (+ 1504us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:41.733166 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:41.733705 (+ 539us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:41.734245 (+ 540us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:41.735684 (+ 1439us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:41.737722 (+ 2038us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:41.738962 (+ 1240us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:41.739606 (+ 644us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:41.742814 (+ 3208us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:41.742922 (+ 108us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:41.742944 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:41.742993 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:41.746710 (+ 3717us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:41.747427 (+ 717us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:41.747435 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:41.747444 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:41.747571 (+ 127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:41.747849 (+ 278us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:41.747854 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:41.747858 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:41.748537 (+ 679us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:41.748801 (+ 264us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:41.749323 (+ 522us) server_negotiation.cc:300] Negotiation successful
1027 14:54:41.749738 (+ 415us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":590,"thread_start_us":370,"threads_started":1}
I20251027 14:54:41.751453 15128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: No bootstrap required, opened a new log
I20251027 14:54:41.757308 15128 raft_consensus.cc:359] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:41.757663 15128 raft_consensus.cc:385] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:41.757757 15128 raft_consensus.cc:740] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625969dad8747ffb083fc47c71eb5bd, State: Initialized, Role: FOLLOWER
I20251027 14:54:41.758325 15128 consensus_queue.cc:260] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:41.758523 15128 raft_consensus.cc:399] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:54:41.758638 15128 raft_consensus.cc:493] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:54:41.758781 15128 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:41.761021 15128 raft_consensus.cc:515] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:41.761502 15128 leader_election.cc:304] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e625969dad8747ffb083fc47c71eb5bd; no voters:
I20251027 14:54:41.762100 15128 leader_election.cc:290] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:54:41.762240 15133 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:41.763289 15133 raft_consensus.cc:697] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 LEADER]: Becoming Leader. State: Replica: e625969dad8747ffb083fc47c71eb5bd, State: Running, Role: LEADER
I20251027 14:54:41.763909 15133 consensus_queue.cc:237] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:41.764164 15128 sys_catalog.cc:565] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:41.766837 15135 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e625969dad8747ffb083fc47c71eb5bd. Latest consensus state: current_term: 1 leader_uuid: "e625969dad8747ffb083fc47c71eb5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } } }
I20251027 14:54:41.767313 15135 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:41.766834 15134 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e625969dad8747ffb083fc47c71eb5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } } }
I20251027 14:54:41.767731 15134 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:41.768218 15142 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:41.774237 15142 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:41.783170 15142 catalog_manager.cc:1357] Generated new cluster ID: d31fc0346fd84120b95b882286b094d6
I20251027 14:54:41.783299 15142 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:54:41.799500 15142 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:54:41.800781 15142 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:54:41.815035 15142 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Generated new TSK 0
I20251027 14:54:41.815973 15142 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:41.911360 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:39727
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:42.246598 15156 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:42.246948 15156 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:42.247063 15156 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:42.255612 15156 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:42.255726 15156 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:42.255887 15156 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:54:42.266896 15156 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:42.268486 15156 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:42.269889 15156 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:42.279600 15161 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:42.279693 15164 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:42.279722 15162 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:42.280982 15156 server_base.cc:1047] running on GCE node
I20251027 14:54:42.281615 15156 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:42.282541 15156 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:42.283761 15156 hybrid_clock.cc:648] HybridClock initialized: now 1761576882283689 us; error 42 us; skew 500 ppm
I20251027 14:54:42.284089 15156 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:42 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:42.287328 15156 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:54:42.288978 15156 webserver.cc:492] Webserver started at http://127.0.104.1:38519/ using document root <none> and password file <none>
I20251027 14:54:42.289700 15156 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:42.289791 15156 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:42.290169 15156 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:42.292886 15156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "edd96d643611465cb7863e4035ce6f22"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "1702941f30bec64bea5af661fbbb15e8"
server_key_iv: "da36bae8087451443a40921ad0069864"
server_key_version: "encryptionkey@0"
I20251027 14:54:42.293664 15156 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "edd96d643611465cb7863e4035ce6f22"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "1702941f30bec64bea5af661fbbb15e8"
server_key_iv: "da36bae8087451443a40921ad0069864"
server_key_version: "encryptionkey@0"
I20251027 14:54:42.299089 15156 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:54:42.302697 15171 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:42.304076 15156 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20251027 14:54:42.304230 15156 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "edd96d643611465cb7863e4035ce6f22"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "1702941f30bec64bea5af661fbbb15e8"
server_key_iv: "da36bae8087451443a40921ad0069864"
server_key_version: "encryptionkey@0"
I20251027 14:54:42.304469 15156 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:42.330384 15156 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:42.334334 15156 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:42.334601 15156 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:42.335605 15156 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:42.337162 15156 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:42.337255 15156 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:42.337356 15156 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:42.337424 15156 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:42.368721 15156 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:32845
I20251027 14:54:42.368777 15283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:32845 every 8 connection(s)
I20251027 14:54:42.370174 15156 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:54:42.373143 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15156
I20251027 14:54:42.373347 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:54:42.374075 416 external_mini_cluster.cc:1468] Setting key 3d28be351a94ec61c070dc4bd1913fc2
Oct 27 14:54:42 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:42.390798 15286 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:42.372886 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:41815 (local address 127.0.104.62:39727)
1027 14:54:42.373348 (+ 462us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:42.373353 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:42.374644 (+ 1291us) server_negotiation.cc:408] Connection header received
1027 14:54:42.376429 (+ 1785us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:42.376436 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:42.376524 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:42.376624 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:42.378462 (+ 1838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.379383 (+ 921us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:42.380554 (+ 1171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.381011 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:42.383944 (+ 2933us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:42.383984 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:42.383989 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:42.384027 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:42.387185 (+ 3158us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:42.388088 (+ 903us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:42.388095 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:42.388118 (+ 23us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:42.388233 (+ 115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:42.388690 (+ 457us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:42.388694 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:42.388697 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:42.388987 (+ 290us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:42.389148 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:42.390139 (+ 991us) server_negotiation.cc:300] Negotiation successful
1027 14:54:42.390454 (+ 315us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":363,"thread_start_us":255,"threads_started":1}
I20251027 14:54:42.391767 15287 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:42.373567 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:41815)
1027 14:54:42.374450 (+ 883us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:42.374491 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:42.376115 (+ 1624us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:42.376931 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:42.376940 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:42.377532 (+ 592us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:42.378264 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:42.378284 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.379552 (+ 1268us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:42.379556 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:42.380408 (+ 852us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:42.380424 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.380703 (+ 279us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:42.381517 (+ 814us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:42.381543 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:42.383723 (+ 2180us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:42.387409 (+ 3686us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:42.387417 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:42.387440 (+ 23us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:42.387931 (+ 491us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:42.388413 (+ 482us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:42.388419 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:42.388423 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:42.388577 (+ 154us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:42.389156 (+ 579us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:42.389164 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:42.389670 (+ 506us) client_negotiation.cc:769] Sending connection context
1027 14:54:42.390029 (+ 359us) client_negotiation.cc:240] Negotiation successful
1027 14:54:42.390545 (+ 516us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":637,"thread_start_us":441,"threads_started":1}
I20251027 14:54:42.393639 15284 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39727
I20251027 14:54:42.394083 15284 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:42.395231 15284 heartbeater.cc:507] Master 127.0.104.62:39727 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
I20251027 14:54:42.398197 15093 ts_manager.cc:194] Registered new tserver with Master: edd96d643611465cb7863e4035ce6f22 (127.0.104.1:32845)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
I20251027 14:54:42.400614 15093 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:41815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:42.467621 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:39727
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:42.794036 15292 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:42.794363 15292 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:42.794502 15292 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:42.802814 15292 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:42.802930 15292 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:42.803107 15292 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:54:42.813655 15292 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:42.815310 15292 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:42.816833 15292 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:42.826335 15297 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:42.826509 15298 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:42.826692 15300 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:42.827600 15292 server_base.cc:1047] running on GCE node
I20251027 14:54:42.828226 15292 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:42.829257 15292 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:42.830488 15292 hybrid_clock.cc:648] HybridClock initialized: now 1761576882830411 us; error 50 us; skew 500 ppm
I20251027 14:54:42.830796 15292 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:42 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:42.834074 15292 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:54:42.835646 15292 webserver.cc:492] Webserver started at http://127.0.104.2:37511/ using document root <none> and password file <none>
I20251027 14:54:42.836429 15292 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:42.836521 15292 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:42.836858 15292 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:42.839655 15292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "e75be72be38a4c32a696bb85ba02f75e"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "86bc93cd3d25c4fc2a088eb8c5f64354"
server_key_iv: "7cbabd74cac04be7c82b4dba551afdff"
server_key_version: "encryptionkey@0"
I20251027 14:54:42.840469 15292 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "e75be72be38a4c32a696bb85ba02f75e"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "86bc93cd3d25c4fc2a088eb8c5f64354"
server_key_iv: "7cbabd74cac04be7c82b4dba551afdff"
server_key_version: "encryptionkey@0"
I20251027 14:54:42.845824 15292 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20251027 14:54:42.849390 15307 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:42.850865 15292 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:54:42.851022 15292 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "e75be72be38a4c32a696bb85ba02f75e"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "86bc93cd3d25c4fc2a088eb8c5f64354"
server_key_iv: "7cbabd74cac04be7c82b4dba551afdff"
server_key_version: "encryptionkey@0"
I20251027 14:54:42.851225 15292 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:42.885372 15292 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:42.889161 15292 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:42.889390 15292 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:42.890345 15292 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:42.891922 15292 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:42.892009 15292 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:42.892094 15292 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:42.892149 15292 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:42.921821 15292 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33565
I20251027 14:54:42.921872 15419 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33565 every 8 connection(s)
I20251027 14:54:42.923156 15292 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:54:42.929785 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15292
I20251027 14:54:42.929994 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:54:42.930368 416 external_mini_cluster.cc:1468] Setting key ac96b9e7170feed60022a492efdc697e
Oct 27 14:54:42 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576882, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:42.941366 15422 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:42.925766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:51853 (local address 127.0.104.62:39727)
1027 14:54:42.926190 (+ 424us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:42.926195 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:42.927097 (+ 902us) server_negotiation.cc:408] Connection header received
1027 14:54:42.928640 (+ 1543us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:42.928648 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:42.928749 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:42.928859 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:42.930616 (+ 1757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.931643 (+ 1027us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:42.932815 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.933258 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:42.936119 (+ 2861us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:42.936155 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:42.936159 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:42.936188 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:42.938610 (+ 2422us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:42.939265 (+ 655us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:42.939270 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:42.939272 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:42.939351 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:42.939651 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:42.939656 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:42.939659 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:42.939898 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:42.940039 (+ 141us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:42.940773 (+ 734us) server_negotiation.cc:300] Negotiation successful
1027 14:54:42.941020 (+ 247us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":328,"thread_start_us":231,"threads_started":1}
I20251027 14:54:42.942337 15423 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:42.926257 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:51853)
1027 14:54:42.926929 (+ 672us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:42.926973 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:42.928306 (+ 1333us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:42.929140 (+ 834us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:42.929153 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:42.929709 (+ 556us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:42.930425 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:42.930447 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.931784 (+ 1337us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:42.931788 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:42.932682 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:42.932694 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:42.932945 (+ 251us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:42.933699 (+ 754us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:42.933722 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:42.935925 (+ 2203us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:42.938746 (+ 2821us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:42.938755 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:42.938774 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:42.939154 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:42.939446 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:42.939450 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:42.939453 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:42.939556 (+ 103us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:42.940024 (+ 468us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:42.940033 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:42.940348 (+ 315us) client_negotiation.cc:769] Sending connection context
1027 14:54:42.940676 (+ 328us) client_negotiation.cc:240] Negotiation successful
1027 14:54:42.941203 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":291,"threads_started":1}
I20251027 14:54:42.943846 15420 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39727
I20251027 14:54:42.944190 15420 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:42.945029 15420 heartbeater.cc:507] Master 127.0.104.62:39727 requested a full tablet report, sending...
I20251027 14:54:42.946632 15093 ts_manager.cc:194] Registered new tserver with Master: e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565)
I20251027 14:54:42.947717 15093 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:51853
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:54:43.016741 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:39727
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:43.336427 15428 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:43.336733 15428 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:43.336841 15428 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:43.344879 15428 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:43.344986 15428 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:43.345151 15428 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:54:43.355540 15428 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:43.357255 15428 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:43.358580 15428 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:43.367835 15434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:43.367816 15433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:43.368613 15436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:43.369407 15428 server_base.cc:1047] running on GCE node
I20251027 14:54:43.369994 15428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:43.370895 15428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:43.372148 15428 hybrid_clock.cc:648] HybridClock initialized: now 1761576883372060 us; error 51 us; skew 500 ppm
I20251027 14:54:43.372478 15428 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:43 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:43.375622 15428 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:54:43.377252 15428 webserver.cc:492] Webserver started at http://127.0.104.3:43967/ using document root <none> and password file <none>
I20251027 14:54:43.377969 15428 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:43.378047 15428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:43.378382 15428 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:54:43.380991 15428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "049bc033049244cc9653fcebb6af78d9"
format_stamp: "Formatted at 2025-10-27 14:54:43 on dist-test-slave-hxxt"
server_key: "3d6ba4f73b6aa6ef032bdf6e719f63f5"
server_key_iv: "93f04e55b26a40da36c848c7f3a69abe"
server_key_version: "encryptionkey@0"
I20251027 14:54:43.381732 15428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "049bc033049244cc9653fcebb6af78d9"
format_stamp: "Formatted at 2025-10-27 14:54:43 on dist-test-slave-hxxt"
server_key: "3d6ba4f73b6aa6ef032bdf6e719f63f5"
server_key_iv: "93f04e55b26a40da36c848c7f3a69abe"
server_key_version: "encryptionkey@0"
I20251027 14:54:43.387085 15428 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:54:43.390504 15443 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:43.391856 15428 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:54:43.392009 15428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "049bc033049244cc9653fcebb6af78d9"
format_stamp: "Formatted at 2025-10-27 14:54:43 on dist-test-slave-hxxt"
server_key: "3d6ba4f73b6aa6ef032bdf6e719f63f5"
server_key_iv: "93f04e55b26a40da36c848c7f3a69abe"
server_key_version: "encryptionkey@0"
I20251027 14:54:43.392199 15428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:43.404745 15284 heartbeater.cc:499] Master 127.0.104.62:39727 was elected leader, sending a full tablet report...
I20251027 14:54:43.414438 15428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:43.418556 15428 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:43.418828 15428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:43.419814 15428 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:43.421363 15428 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:43.421451 15428 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:43.421555 15428 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:43.421620 15428 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:43.452236 15428 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:38427
I20251027 14:54:43.452282 15555 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:38427 every 8 connection(s)
I20251027 14:54:43.453655 15428 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:54:43.457639 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15428
I20251027 14:54:43.457845 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:54:43.458302 416 external_mini_cluster.cc:1468] Setting key 17418edd11408cc52901f5445bb549df
Oct 27 14:54:43 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576883, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:43.472733 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.456304 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:34967 (local address 127.0.104.62:39727)
1027 14:54:43.456749 (+ 445us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.456754 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.457887 (+ 1133us) server_negotiation.cc:408] Connection header received
1027 14:54:43.459729 (+ 1842us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.459735 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.459825 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.459927 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:43.461946 (+ 2019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.462890 (+ 944us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.463958 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.464428 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.467206 (+ 2778us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:43.467244 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:43.467248 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:43.467282 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:43.469808 (+ 2526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.470504 (+ 696us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.470509 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.470511 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.470580 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.470903 (+ 323us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.470909 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.470912 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.471172 (+ 260us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:43.471305 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.472065 (+ 760us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.472334 (+ 269us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":214,"threads_started":1}
I20251027 14:54:43.473796 15559 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.456900 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:34967)
1027 14:54:43.457705 (+ 805us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:43.457761 (+ 56us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:43.459425 (+ 1664us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:43.460197 (+ 772us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:43.460206 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:43.461032 (+ 826us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:43.461782 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.461801 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.463044 (+ 1243us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.463049 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:43.463824 (+ 775us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.463836 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.464104 (+ 268us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.464817 (+ 713us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:43.464843 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:43.466997 (+ 2154us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:43.469962 (+ 2965us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:43.469972 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:43.469987 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:43.470378 (+ 391us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:43.470673 (+ 295us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:43.470678 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:43.470681 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:43.470795 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:43.471293 (+ 498us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:43.471301 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:43.471638 (+ 337us) client_negotiation.cc:769] Sending connection context
1027 14:54:43.471953 (+ 315us) client_negotiation.cc:240] Negotiation successful
1027 14:54:43.472506 (+ 553us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":525,"thread_start_us":330,"threads_started":1}
I20251027 14:54:43.475106 15556 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39727
I20251027 14:54:43.475446 15556 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:43.476266 15556 heartbeater.cc:507] Master 127.0.104.62:39727 requested a full tablet report, sending...
I20251027 14:54:43.477730 15093 ts_manager.cc:194] Registered new tserver with Master: 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427)
I20251027 14:54:43.478543 15093 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:34967
I20251027 14:54:43.486235 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:54:43.502640 15558 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.490838 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42704 (local address 127.0.104.62:39727)
1027 14:54:43.491011 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.491016 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.491415 (+ 399us) server_negotiation.cc:408] Connection header received
1027 14:54:43.491555 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.491560 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.491631 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.491707 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:43.493350 (+ 1643us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.494260 (+ 910us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.495483 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.495845 (+ 362us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.497625 (+ 1780us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:43.497692 (+ 67us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:43.497697 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:43.497729 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:43.500491 (+ 2762us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.501071 (+ 580us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.501075 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.501077 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.501142 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.501466 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.501470 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.501472 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.501762 (+ 290us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:43.501944 (+ 182us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.502203 (+ 259us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.502393 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:54:43.506209 15093 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42704:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:54:43.511806 15093 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:54:43.531924 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.523685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:38427 (local address 127.0.0.1:55638)
1027 14:54:43.524417 (+ 732us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:43.524465 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:43.524640 (+ 175us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:43.526266 (+ 1626us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:43.526273 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:43.526303 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:43.527083 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.527098 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.529311 (+ 2213us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.529317 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:43.531013 (+ 1696us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.531028 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.531196 (+ 168us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.531256 (+ 60us) client_negotiation.cc:769] Sending connection context
1027 14:54:43.531398 (+ 142us) client_negotiation.cc:240] Negotiation successful
1027 14:54:43.531572 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":481,"thread_start_us":340,"threads_started":1}
I20251027 14:54:43.534184 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.523941 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55638 (local address 127.0.104.3:38427)
1027 14:54:43.525321 (+ 1380us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.525331 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.525362 (+ 31us) server_negotiation.cc:408] Connection header received
1027 14:54:43.525462 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.525469 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.525719 (+ 250us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.525938 (+ 219us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:43.527271 (+ 1333us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.529092 (+ 1821us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.532291 (+ 3199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.533442 (+ 1151us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.533603 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.533723 (+ 120us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.533914 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1229,"thread_start_us":1583,"threads_started":1}
I20251027 14:54:43.534926 15572 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.524137 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54776)
1027 14:54:43.526034 (+ 1897us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:43.526053 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:43.526166 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:43.527172 (+ 1006us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:43.527176 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:43.527203 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:43.528928 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.528943 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.532606 (+ 3663us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.532612 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:43.534245 (+ 1633us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.534260 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.534399 (+ 139us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.534422 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:54:43.534474 (+ 52us) client_negotiation.cc:240] Negotiation successful
1027 14:54:43.534588 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1787,"mutex_wait_us":681,"thread_start_us":953,"threads_started":1}
I20251027 14:54:43.537276 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.524575 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54776 (local address 127.0.104.1:32845)
1027 14:54:43.526619 (+ 2044us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.526628 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.526653 (+ 25us) server_negotiation.cc:408] Connection header received
1027 14:54:43.526753 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.526761 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.526958 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.529155 (+ 2197us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:43.529491 (+ 336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.532350 (+ 2859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.535517 (+ 3167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.536590 (+ 1073us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.536728 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.536853 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.537023 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1909,"thread_start_us":1793,"threads_started":1}
I20251027 14:54:43.540325 15491 tablet_service.cc:1505] Processing CreateTablet for tablet 76543cfb23e3438c82f192523f91a2ec (DEFAULT_TABLE table=test-table [id=4a3fb4a71b4e469aa1b3a3f8aa374f7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:43.542184 15491 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76543cfb23e3438c82f192523f91a2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:43.541817 15219 tablet_service.cc:1505] Processing CreateTablet for tablet 76543cfb23e3438c82f192523f91a2ec (DEFAULT_TABLE table=test-table [id=4a3fb4a71b4e469aa1b3a3f8aa374f7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:43.543498 15219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76543cfb23e3438c82f192523f91a2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:43.550060 15569 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.526886 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33565 (local address 127.0.0.1:54474)
1027 14:54:43.534849 (+ 7963us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:43.534867 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:43.534972 (+ 105us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:43.544888 (+ 9916us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:43.544893 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:43.544923 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:43.545362 (+ 439us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.545378 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.547530 (+ 2152us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.547537 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:43.549385 (+ 1848us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.549402 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.549545 (+ 143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.549573 (+ 28us) client_negotiation.cc:769] Sending connection context
1027 14:54:43.549636 (+ 63us) client_negotiation.cc:240] Negotiation successful
1027 14:54:43.549769 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":7869,"thread_start_us":17320,"threads_started":1}
I20251027 14:54:43.551385 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.527117 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54474 (local address 127.0.104.2:33565)
1027 14:54:43.544253 (+ 17136us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.544262 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.544290 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:54:43.544424 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.544432 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.544662 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.544880 (+ 218us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:43.545532 (+ 652us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.547354 (+ 1822us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.549560 (+ 2206us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.550659 (+ 1099us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.550803 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.550943 (+ 140us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.551127 (+ 184us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16990,"thread_start_us":16882,"threads_started":1}
I20251027 14:54:43.555083 15575 tablet_bootstrap.cc:492] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Bootstrap starting.
I20251027 14:54:43.557360 15576 tablet_bootstrap.cc:492] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Bootstrap starting.
I20251027 14:54:43.557593 15355 tablet_service.cc:1505] Processing CreateTablet for tablet 76543cfb23e3438c82f192523f91a2ec (DEFAULT_TABLE table=test-table [id=4a3fb4a71b4e469aa1b3a3f8aa374f7b]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:43.558902 15575 tablet_bootstrap.cc:654] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:43.558933 15355 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 76543cfb23e3438c82f192523f91a2ec. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:43.560240 15575 log.cc:826] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:43.560832 15576 tablet_bootstrap.cc:654] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:43.561971 15576 log.cc:826] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:43.563400 15575 tablet_bootstrap.cc:492] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: No bootstrap required, opened a new log
I20251027 14:54:43.563686 15575 ts_tablet_manager.cc:1403] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20251027 14:54:43.565236 15576 tablet_bootstrap.cc:492] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: No bootstrap required, opened a new log
I20251027 14:54:43.565605 15576 ts_tablet_manager.cc:1403] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20251027 14:54:43.570714 15579 tablet_bootstrap.cc:492] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Bootstrap starting.
I20251027 14:54:43.570858 15575 raft_consensus.cc:359] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.571681 15575 raft_consensus.cc:385] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:43.571826 15575 raft_consensus.cc:740] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd96d643611465cb7863e4035ce6f22, State: Initialized, Role: FOLLOWER
I20251027 14:54:43.572770 15575 consensus_queue.cc:260] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.572675 15576 raft_consensus.cc:359] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.573042 15576 raft_consensus.cc:385] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:43.573145 15576 raft_consensus.cc:740] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049bc033049244cc9653fcebb6af78d9, State: Initialized, Role: FOLLOWER
I20251027 14:54:43.573251 15580 raft_consensus.cc:493] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:43.573491 15580 raft_consensus.cc:515] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.574054 15576 consensus_queue.cc:260] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.574297 15579 tablet_bootstrap.cc:654] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:43.575639 15580 leader_election.cc:290] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427), e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565)
I20251027 14:54:43.580209 15575 ts_tablet_manager.cc:1434] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Time spent starting tablet: real 0.016s user 0.011s sys 0.000s
I20251027 14:54:43.581337 15556 heartbeater.cc:499] Master 127.0.104.62:39727 was elected leader, sending a full tablet report...
I20251027 14:54:43.582306 15576 ts_tablet_manager.cc:1434] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Time spent starting tablet: real 0.017s user 0.013s sys 0.000s
I20251027 14:54:43.584095 15579 log.cc:826] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:43.588248 15583 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.575874 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33565 (local address 127.0.104.1:56877)
1027 14:54:43.579799 (+ 3925us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:43.579819 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:43.580492 (+ 673us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:43.582662 (+ 2170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:43.582667 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:43.582699 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:43.583544 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.583561 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.585410 (+ 1849us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.585415 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:43.587414 (+ 1999us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.587431 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.587627 (+ 196us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.587658 (+ 31us) client_negotiation.cc:769] Sending connection context
1027 14:54:43.587728 (+ 70us) client_negotiation.cc:240] Negotiation successful
1027 14:54:43.587892 (+ 164us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3806,"thread_start_us":3687,"threads_started":1}
I20251027 14:54:43.589398 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.576295 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:56877 (local address 127.0.104.2:33565)
1027 14:54:43.576483 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.576490 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.579915 (+ 3425us) server_negotiation.cc:408] Connection header received
1027 14:54:43.580858 (+ 943us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.580865 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.580948 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.581012 (+ 64us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:43.583750 (+ 2738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.585159 (+ 1409us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.587608 (+ 2449us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.588834 (+ 1226us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.588893 (+ 59us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.588979 (+ 86us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.589146 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:54:43.591722 15584 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.576308 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:38427 (local address 127.0.104.1:53731)
1027 14:54:43.585212 (+ 8904us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:43.585231 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:43.585357 (+ 126us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:43.585762 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:43.585767 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:43.585797 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:43.586619 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.586635 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.588302 (+ 1667us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.588307 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:43.589956 (+ 1649us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:43.589973 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.591168 (+ 1195us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.591197 (+ 29us) client_negotiation.cc:769] Sending connection context
1027 14:54:43.591278 (+ 81us) client_negotiation.cc:240] Negotiation successful
1027 14:54:43.591415 (+ 137us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":8793,"thread_start_us":8614,"threads_started":1}
I20251027 14:54:43.591729 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.577298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:53731 (local address 127.0.104.3:38427)
1027 14:54:43.577485 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.577491 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.585323 (+ 7832us) server_negotiation.cc:408] Connection header received
1027 14:54:43.585557 (+ 234us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.585562 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.585635 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.585721 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:43.586822 (+ 1101us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.588038 (+ 1216us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.590156 (+ 2118us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.591248 (+ 1092us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.591301 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.591377 (+ 76us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.591498 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:54:43.592805 15579 tablet_bootstrap.cc:492] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: No bootstrap required, opened a new log
I20251027 14:54:43.593096 15579 ts_tablet_manager.cc:1403] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Time spent bootstrapping tablet: real 0.023s user 0.010s sys 0.002s
I20251027 14:54:43.590588 15375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76543cfb23e3438c82f192523f91a2ec" candidate_uuid: "edd96d643611465cb7863e4035ce6f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75be72be38a4c32a696bb85ba02f75e" is_pre_election: true
I20251027 14:54:43.593137 15511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76543cfb23e3438c82f192523f91a2ec" candidate_uuid: "edd96d643611465cb7863e4035ce6f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049bc033049244cc9653fcebb6af78d9" is_pre_election: true
I20251027 14:54:43.593639 15511 raft_consensus.cc:2468] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate edd96d643611465cb7863e4035ce6f22 in term 0.
I20251027 14:54:43.594288 15172 leader_election.cc:304] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049bc033049244cc9653fcebb6af78d9, edd96d643611465cb7863e4035ce6f22; no voters:
W20251027 14:54:43.594599 15175 leader_election.cc:343] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): Illegal state: must be running to vote when last-logged opid is not known
I20251027 14:54:43.594691 15580 raft_consensus.cc:2804] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:54:43.594843 15580 raft_consensus.cc:493] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:54:43.594959 15580 raft_consensus.cc:3060] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:43.597081 15579 raft_consensus.cc:359] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.597395 15579 raft_consensus.cc:385] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:43.597378 15580 raft_consensus.cc:515] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.597496 15579 raft_consensus.cc:740] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75be72be38a4c32a696bb85ba02f75e, State: Initialized, Role: FOLLOWER
I20251027 14:54:43.597991 15580 leader_election.cc:290] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 1 election: Requested vote from peers 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427), e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565)
I20251027 14:54:43.598194 15579 consensus_queue.cc:260] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.598393 15511 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76543cfb23e3438c82f192523f91a2ec" candidate_uuid: "edd96d643611465cb7863e4035ce6f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "049bc033049244cc9653fcebb6af78d9"
I20251027 14:54:43.598532 15375 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "76543cfb23e3438c82f192523f91a2ec" candidate_uuid: "edd96d643611465cb7863e4035ce6f22" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75be72be38a4c32a696bb85ba02f75e"
I20251027 14:54:43.598590 15511 raft_consensus.cc:3060] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:43.598850 15375 raft_consensus.cc:3060] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:43.599306 15420 heartbeater.cc:499] Master 127.0.104.62:39727 was elected leader, sending a full tablet report...
I20251027 14:54:43.599815 15579 ts_tablet_manager.cc:1434] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Time spent starting tablet: real 0.007s user 0.005s sys 0.002s
I20251027 14:54:43.602010 15511 raft_consensus.cc:2468] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd96d643611465cb7863e4035ce6f22 in term 1.
I20251027 14:54:43.602435 15172 leader_election.cc:304] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049bc033049244cc9653fcebb6af78d9, edd96d643611465cb7863e4035ce6f22; no voters:
I20251027 14:54:43.602732 15580 raft_consensus.cc:2804] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:43.602794 15375 raft_consensus.cc:2468] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate edd96d643611465cb7863e4035ce6f22 in term 1.
I20251027 14:54:43.602906 15580 raft_consensus.cc:697] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 1 LEADER]: Becoming Leader. State: Replica: edd96d643611465cb7863e4035ce6f22, State: Running, Role: LEADER
I20251027 14:54:43.603330 15580 consensus_queue.cc:237] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } }
I20251027 14:54:43.608516 15093 catalog_manager.cc:5649] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 reported cstate change: term changed from 0 to 1, leader changed from <none> to edd96d643611465cb7863e4035ce6f22 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "edd96d643611465cb7863e4035ce6f22" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } health_report { overall_health: UNKNOWN } } }
W20251027 14:54:43.622521 15285 tablet.cc:2378] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:54:43.638316 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.632862 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54780 (local address 127.0.104.1:32845)
1027 14:54:43.633070 (+ 208us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.633076 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.633094 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:43.633264 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.633269 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.633347 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.633477 (+ 130us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:54:43.634062 (+ 585us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.635304 (+ 1242us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.636610 (+ 1306us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.637019 (+ 409us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.637195 (+ 176us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:54:43.637727 (+ 532us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:54:43.637865 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.637965 (+ 100us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.638083 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251027 14:54:43.646131 15375 raft_consensus.cc:1275] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Refusing update from remote peer edd96d643611465cb7863e4035ce6f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:43.646178 15511 raft_consensus.cc:1275] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Refusing update from remote peer edd96d643611465cb7863e4035ce6f22: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:43.647002 15581 consensus_queue.cc:1048] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:43.647260 15580 consensus_queue.cc:1048] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [LEADER]: Connected to new peer: Peer: permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:43.665220 15590 mvcc.cc:204] Tried to move back new op lower bound from 7215418915405131776 to 7215418915243909120. Current Snapshot: MvccSnapshot[applied={T|T < 7215418915405131776}]
I20251027 14:54:43.674233 15093 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42704:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:54:43.674463 15093 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42704:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:54:43.677773 15093 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Sending DeleteTablet for 3 replicas of tablet 76543cfb23e3438c82f192523f91a2ec
I20251027 14:54:43.679011 15491 tablet_service.cc:1552] Processing DeleteTablet for tablet 76543cfb23e3438c82f192523f91a2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:43 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:55638
I20251027 14:54:43.679024 15355 tablet_service.cc:1552] Processing DeleteTablet for tablet 76543cfb23e3438c82f192523f91a2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:43 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:54474
I20251027 14:54:43.679817 15219 tablet_service.cc:1552] Processing DeleteTablet for tablet 76543cfb23e3438c82f192523f91a2ec with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:43 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:54776
I20251027 14:54:43.681160 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15156
I20251027 14:54:43.689244 15600 tablet_replica.cc:333] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: stopping tablet replica
I20251027 14:54:43.689865 15600 raft_consensus.cc:2243] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:54:43.690686 15600 raft_consensus.cc:2272] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:43.698041 15599 tablet_replica.cc:333] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: stopping tablet replica
I20251027 14:54:43.698558 15599 raft_consensus.cc:2243] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:54:43.699107 15599 raft_consensus.cc:2272] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:43.700068 15598 tablet_replica.cc:333] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: stopping tablet replica
I20251027 14:54:43.700866 15598 raft_consensus.cc:2243] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:54:43.701315 15600 ts_tablet_manager.cc:1916] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:43.701409 15598 raft_consensus.cc:2272] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:43.702685 15599 ts_tablet_manager.cc:1916] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Deleting tablet data with delete state TABLET_DATA_DELETED
Oct 27 14:54:43 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:54:43.708619 15598 ts_tablet_manager.cc:1916] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:43.709568 15600 ts_tablet_manager.cc:1929] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:43.709764 15600 log.cc:1199] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/76543cfb23e3438c82f192523f91a2ec
I20251027 14:54:43.710299 15600 ts_tablet_manager.cc:1950] T 76543cfb23e3438c82f192523f91a2ec P edd96d643611465cb7863e4035ce6f22: Deleting consensus metadata
I20251027 14:54:43.712188 15078 catalog_manager.cc:4985] TS edd96d643611465cb7863e4035ce6f22 (127.0.104.1:32845): tablet 76543cfb23e3438c82f192523f91a2ec (table test-table [id=4a3fb4a71b4e469aa1b3a3f8aa374f7b]) successfully deleted
I20251027 14:54:43.713423 15571 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.682450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54782 (local address 127.0.104.1:32845)
1027 14:54:43.682633 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.682640 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.691648 (+ 9008us) server_negotiation.cc:408] Connection header received
1027 14:54:43.691854 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.691861 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.691950 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.692075 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:43.694566 (+ 2491us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.696074 (+ 1508us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.698075 (+ 2001us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.698550 (+ 475us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.705302 (+ 6752us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:43.705373 (+ 71us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:43.705391 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:43.705440 (+ 49us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:43.709858 (+ 4418us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.710775 (+ 917us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.710784 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.710787 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.710887 (+ 100us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.711294 (+ 407us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.711299 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.711302 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.711647 (+ 345us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:43.712118 (+ 471us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.712789 (+ 671us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.713046 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:54:43.715135 15279 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:43.716140 15599 ts_tablet_manager.cc:1929] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:43.716360 15599 log.cc:1199] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/76543cfb23e3438c82f192523f91a2ec
I20251027 14:54:43.716495 15598 ts_tablet_manager.cc:1929] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:43.716758 15598 log.cc:1199] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/76543cfb23e3438c82f192523f91a2ec
I20251027 14:54:43.717133 15599 ts_tablet_manager.cc:1950] T 76543cfb23e3438c82f192523f91a2ec P 049bc033049244cc9653fcebb6af78d9: Deleting consensus metadata
I20251027 14:54:43.717454 15598 ts_tablet_manager.cc:1950] T 76543cfb23e3438c82f192523f91a2ec P e75be72be38a4c32a696bb85ba02f75e: Deleting consensus metadata
I20251027 14:54:43.719362 15077 catalog_manager.cc:4985] TS 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427): tablet 76543cfb23e3438c82f192523f91a2ec (table test-table [id=4a3fb4a71b4e469aa1b3a3f8aa374f7b]) successfully deleted
I20251027 14:54:43.719532 15080 catalog_manager.cc:4985] TS e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): tablet 76543cfb23e3438c82f192523f91a2ec (table test-table [id=4a3fb4a71b4e469aa1b3a3f8aa374f7b]) successfully deleted
I20251027 14:54:43.813933 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15156
I20251027 14:54:43.829434 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15292
Oct 27 14:54:43 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:54:43.843319 15574 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.830288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54490 (local address 127.0.104.2:33565)
1027 14:54:43.830452 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.830458 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.830481 (+ 23us) server_negotiation.cc:408] Connection header received
1027 14:54:43.830594 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.830601 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.830731 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.830849 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:43.832307 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.833226 (+ 919us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.834383 (+ 1157us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.834753 (+ 370us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.838072 (+ 3319us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:43.838121 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:43.838137 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:43.838170 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:43.841149 (+ 2979us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.841699 (+ 550us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.841707 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.841709 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.841801 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.842096 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.842101 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.842103 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.842401 (+ 298us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:43.842695 (+ 294us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.842806 (+ 111us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.842994 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:54:43.843954 15415 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:43.933315 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15292
I20251027 14:54:43.947350 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15428
Oct 27 14:54:43 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576881, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:54:43.967163 15570 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:43.948050 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55652 (local address 127.0.104.3:38427)
1027 14:54:43.948192 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:43.948196 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:43.948212 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:54:43.948362 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:43.948366 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:43.948469 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:43.948552 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:43.950120 (+ 1568us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.950994 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:43.952175 (+ 1181us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:43.952697 (+ 522us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:43.961805 (+ 9108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:43.961849 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:43.961866 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:43.961907 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:43.964556 (+ 2649us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.965242 (+ 686us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.965252 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.965255 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.965368 (+ 113us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:43.965678 (+ 310us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:43.965684 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:43.965687 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:43.965974 (+ 287us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:43.966263 (+ 289us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:43.966521 (+ 258us) server_negotiation.cc:300] Negotiation successful
1027 14:54:43.966822 (+ 301us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:54:43.967819 15551 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:44.057520 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15428
I20251027 14:54:44.071601 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15061
I20251027 14:54:44.072736 15123 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:44.182144 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15061
I20251027 14:54:44.200441 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39727
--webserver_interface=127.0.104.62
--webserver_port=36661
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--ipki_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:44.531318 15606 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:44.531630 15606 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:44.531765 15606 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:44.539901 15606 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:44.540002 15606 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:44.540086 15606 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:44.540149 15606 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:44.540231 15606 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:44.550668 15606 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_private_key_password_cmd=echo foo
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:39727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=36661
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:44.552234 15606 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:44.553593 15606 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:44.571689 15612 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:44.571702 15613 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:44.571766 15615 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:44.572614 15606 server_base.cc:1047] running on GCE node
I20251027 14:54:44.573447 15606 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:44.575040 15606 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:44.576262 15606 hybrid_clock.cc:648] HybridClock initialized: now 1761576884576193 us; error 40 us; skew 500 ppm
I20251027 14:54:44.576706 15606 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:44 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576884, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:44.580474 15606 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:44.582304 15606 webserver.cc:492] Webserver started at http://127.0.104.62:36661/ using document root <none> and password file <none>
I20251027 14:54:44.583148 15606 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:44.583249 15606 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:44.589696 15606 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20251027 14:54:44.592821 15622 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:44.594295 15606 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20251027 14:54:44.594478 15606 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "e625969dad8747ffb083fc47c71eb5bd"
format_stamp: "Formatted at 2025-10-27 14:54:41 on dist-test-slave-hxxt"
server_key: "01a4d8b63c49655046550a6011f8b748"
server_key_iv: "aab154fc1c62670a4107b7abdbd61c94"
server_key_version: "encryptionkey@0"
I20251027 14:54:44.595309 15606 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:44.612643 15606 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:44.616714 15606 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:44.617007 15606 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:44.636089 15606 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39727
I20251027 14:54:44.636111 15673 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39727 every 8 connection(s)
I20251027 14:54:44.637696 15606 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:44.643085 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15606
I20251027 14:54:44.643914 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:32845
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:39727
--webserver_port=38519
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:44.645332 15674 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:54:44.648612 15674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap starting.
I20251027 14:54:44.666805 15674 log.cc:826] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:44.679723 15674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:54:44.680222 15674 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap complete.
I20251027 14:54:44.686945 15674 raft_consensus.cc:359] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:44.687613 15674 raft_consensus.cc:740] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625969dad8747ffb083fc47c71eb5bd, State: Initialized, Role: FOLLOWER
I20251027 14:54:44.688282 15674 consensus_queue.cc:260] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:44.688526 15674 raft_consensus.cc:399] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:54:44.688674 15674 raft_consensus.cc:493] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:54:44.688834 15674 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 1 FOLLOWER]: Advancing to term 2
I20251027 14:54:44.692536 15674 raft_consensus.cc:515] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:44.693010 15674 leader_election.cc:304] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e625969dad8747ffb083fc47c71eb5bd; no voters:
I20251027 14:54:44.693608 15674 leader_election.cc:290] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [CANDIDATE]: Term 2 election: Requested vote from peers
I20251027 14:54:44.693751 15678 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Leader election won for term 2
I20251027 14:54:44.694754 15678 raft_consensus.cc:697] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 LEADER]: Becoming Leader. State: Replica: e625969dad8747ffb083fc47c71eb5bd, State: Running, Role: LEADER
I20251027 14:54:44.695343 15678 consensus_queue.cc:237] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:44.695761 15674 sys_catalog.cc:565] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:44.697781 15680 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e625969dad8747ffb083fc47c71eb5bd. Latest consensus state: current_term: 2 leader_uuid: "e625969dad8747ffb083fc47c71eb5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } } }
I20251027 14:54:44.698348 15679 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "e625969dad8747ffb083fc47c71eb5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } } }
I20251027 14:54:44.698906 15680 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:44.699242 15679 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:44.704975 15684 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:44.713461 15684 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:44.715054 15684 catalog_manager.cc:1269] Loaded cluster ID: d31fc0346fd84120b95b882286b094d6
I20251027 14:54:44.715237 15684 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:54:44.718335 15684 catalog_manager.cc:1324] IPKI private key was stored in cleartext, attempting to encrypt it.
I20251027 14:54:44.722460 15684 catalog_manager.cc:1334] IPKI private key successfully encrypted.
I20251027 14:54:44.735628 15684 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:54:44.737537 15684 catalog_manager.cc:6033] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Loaded TSK: 0
I20251027 14:54:44.739559 15684 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251027 14:54:44.984546 15676 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:44.984845 15676 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:44.984968 15676 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:44.993294 15676 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:44.993391 15676 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:44.993573 15676 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:54:45.004173 15676 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:32845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=38519
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:45.005808 15676 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:45.007179 15676 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:45.016726 15701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:45.016252 15702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:45.018461 15704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:45.019007 15676 server_base.cc:1047] running on GCE node
I20251027 14:54:45.019568 15676 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:45.020510 15676 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:45.021726 15676 hybrid_clock.cc:648] HybridClock initialized: now 1761576885021648 us; error 59 us; skew 500 ppm
I20251027 14:54:45.022012 15676 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:45 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:45.025059 15676 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:54:45.026679 15676 webserver.cc:492] Webserver started at http://127.0.104.1:38519/ using document root <none> and password file <none>
I20251027 14:54:45.027388 15676 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:45.027477 15676 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:45.032891 15676 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:54:45.035612 15711 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.037110 15676 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:45.037261 15676 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "edd96d643611465cb7863e4035ce6f22"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "1702941f30bec64bea5af661fbbb15e8"
server_key_iv: "da36bae8087451443a40921ad0069864"
server_key_version: "encryptionkey@0"
I20251027 14:54:45.037936 15676 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:45.069612 15676 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:45.073661 15676 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:45.073944 15676 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:45.074962 15676 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:45.076537 15676 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:45.076622 15676 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.076740 15676 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:45.076790 15676 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.106959 15676 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:32845
I20251027 14:54:45.106992 15823 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:32845 every 8 connection(s)
I20251027 14:54:45.108321 15676 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:54:45.115847 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15676
I20251027 14:54:45.116698 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:33565
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:39727
--webserver_port=37511
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:54:45 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:45.129932 15827 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:45.111539 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:53305)
1027 14:54:45.112272 (+ 733us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:45.112312 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:45.113654 (+ 1342us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:45.114919 (+ 1265us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:45.114927 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:45.115566 (+ 639us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:45.116503 (+ 937us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:45.116529 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.118246 (+ 1717us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:45.118252 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:45.119356 (+ 1104us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:45.119371 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.119620 (+ 249us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:45.120352 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:45.120405 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:45.122762 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:45.125995 (+ 3233us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:45.126004 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:45.126019 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:45.126386 (+ 367us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:45.126758 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:45.126762 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:45.126764 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:45.126878 (+ 114us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:45.127497 (+ 619us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:45.127506 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:45.127802 (+ 296us) client_negotiation.cc:769] Sending connection context
1027 14:54:45.128093 (+ 291us) client_negotiation.cc:240] Negotiation successful
1027 14:54:45.128616 (+ 523us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":324,"threads_started":1}
I20251027 14:54:45.129926 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:45.111264 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:53305 (local address 127.0.104.62:39727)
1027 14:54:45.111776 (+ 512us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:45.111788 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:45.112452 (+ 664us) server_negotiation.cc:408] Connection header received
1027 14:54:45.114124 (+ 1672us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:45.114137 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:45.114457 (+ 320us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:45.114813 (+ 356us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:45.116725 (+ 1912us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.118083 (+ 1358us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:45.119504 (+ 1421us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.119891 (+ 387us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:45.122999 (+ 3108us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:45.123055 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:45.123071 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:45.123114 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:45.125853 (+ 2739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:45.126523 (+ 670us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:45.126530 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:45.126536 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:45.126655 (+ 119us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:45.126979 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:45.126983 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:45.126985 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:45.127371 (+ 386us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:45.127542 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:45.128355 (+ 813us) server_negotiation.cc:300] Negotiation successful
1027 14:54:45.128795 (+ 440us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":339,"thread_start_us":226,"threads_started":1}
I20251027 14:54:45.132716 15824 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39727
I20251027 14:54:45.133138 15824 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:45.134133 15824 heartbeater.cc:507] Master 127.0.104.62:39727 requested a full tablet report, sending...
I20251027 14:54:45.136503 15639 ts_manager.cc:194] Registered new tserver with Master: edd96d643611465cb7863e4035ce6f22 (127.0.104.1:32845)
I20251027 14:54:45.138096 15639 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:53305
W20251027 14:54:45.439303 15828 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:45.439581 15828 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:45.439723 15828 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:45.447649 15828 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:45.447747 15828 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:45.447939 15828 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:54:45.458245 15828 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:33565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=37511
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:45.459793 15828 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:45.461109 15828 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:45.469878 15833 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:45.469915 15834 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:45.470362 15836 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:45.470800 15828 server_base.cc:1047] running on GCE node
I20251027 14:54:45.471360 15828 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:45.472270 15828 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:45.473544 15828 hybrid_clock.cc:648] HybridClock initialized: now 1761576885473430 us; error 47 us; skew 500 ppm
I20251027 14:54:45.473841 15828 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:45 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:45.477078 15828 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:54:45.478633 15828 webserver.cc:492] Webserver started at http://127.0.104.2:37511/ using document root <none> and password file <none>
I20251027 14:54:45.479372 15828 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:45.479472 15828 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:45.485064 15828 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251027 14:54:45.487746 15843 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.489135 15828 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:45.489287 15828 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "e75be72be38a4c32a696bb85ba02f75e"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "86bc93cd3d25c4fc2a088eb8c5f64354"
server_key_iv: "7cbabd74cac04be7c82b4dba551afdff"
server_key_version: "encryptionkey@0"
I20251027 14:54:45.489974 15828 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:45.514221 15828 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:45.518911 15828 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:45.519151 15828 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:45.520073 15828 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:45.521552 15828 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:45.521629 15828 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.521715 15828 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:45.521756 15828 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.550993 15828 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33565
I20251027 14:54:45.551051 15955 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33565 every 8 connection(s)
I20251027 14:54:45.552490 15828 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:54:45.558616 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15828
I20251027 14:54:45.559227 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:38427
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:39727
--webserver_port=43967
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:54:45 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:45.571445 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:45.555255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:36297 (local address 127.0.104.62:39727)
1027 14:54:45.555460 (+ 205us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:45.555465 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:45.556691 (+ 1226us) server_negotiation.cc:408] Connection header received
1027 14:54:45.558174 (+ 1483us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:45.558179 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:45.558259 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:45.558367 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:45.560117 (+ 1750us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.561344 (+ 1227us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:45.562523 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.562967 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:45.565851 (+ 2884us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:45.565882 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:45.565886 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:45.565920 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:45.568359 (+ 2439us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:45.569237 (+ 878us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:45.569241 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:45.569244 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:45.569313 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:45.569638 (+ 325us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:45.569642 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:45.569644 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:45.569876 (+ 232us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:45.569985 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:45.570872 (+ 887us) server_negotiation.cc:300] Negotiation successful
1027 14:54:45.571089 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":78}
I20251027 14:54:45.572760 15958 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:45.555720 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:36297)
1027 14:54:45.556515 (+ 795us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:45.556566 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:45.557913 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:45.558583 (+ 670us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:45.558592 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:45.559182 (+ 590us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:45.559940 (+ 758us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:45.559960 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.561553 (+ 1593us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:45.561558 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:45.562387 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:45.562398 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:45.562622 (+ 224us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:45.563268 (+ 646us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:45.563293 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:45.565602 (+ 2309us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:45.568556 (+ 2954us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:45.568566 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:45.568585 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:45.569139 (+ 554us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:45.569399 (+ 260us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:45.569404 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:45.569407 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:45.569552 (+ 145us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:45.569978 (+ 426us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:45.569985 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:45.570394 (+ 409us) client_negotiation.cc:769] Sending connection context
1027 14:54:45.570799 (+ 405us) client_negotiation.cc:240] Negotiation successful
1027 14:54:45.571297 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":503,"thread_start_us":319,"threads_started":1}
I20251027 14:54:45.574286 15956 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39727
I20251027 14:54:45.574713 15956 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:45.575773 15956 heartbeater.cc:507] Master 127.0.104.62:39727 requested a full tablet report, sending...
I20251027 14:54:45.577538 15639 ts_manager.cc:194] Registered new tserver with Master: e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565)
I20251027 14:54:45.578362 15639 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:36297
W20251027 14:54:45.885000 15959 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:45.885305 15959 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:45.885437 15959 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:45.893674 15959 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:45.893780 15959 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:45.893966 15959 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:54:45.904561 15959 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:38427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=43967
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:45.906240 15959 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:45.907598 15959 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:45.916775 15964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:45.916767 15965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:45.917371 15967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:45.917944 15959 server_base.cc:1047] running on GCE node
I20251027 14:54:45.918519 15959 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:45.919423 15959 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:45.920639 15959 hybrid_clock.cc:648] HybridClock initialized: now 1761576885920568 us; error 44 us; skew 500 ppm
I20251027 14:54:45.920933 15959 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:45 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:45.924139 15959 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:54:45.925683 15959 webserver.cc:492] Webserver started at http://127.0.104.3:43967/ using document root <none> and password file <none>
I20251027 14:54:45.926380 15959 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:45.926482 15959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:45.931862 15959 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:54:45.934650 15974 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.936002 15959 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:54:45.936142 15959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "049bc033049244cc9653fcebb6af78d9"
format_stamp: "Formatted at 2025-10-27 14:54:43 on dist-test-slave-hxxt"
server_key: "3d6ba4f73b6aa6ef032bdf6e719f63f5"
server_key_iv: "93f04e55b26a40da36c848c7f3a69abe"
server_key_version: "encryptionkey@0"
I20251027 14:54:45.936864 15959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:45.956735 15959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:45.960611 15959 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:45.960848 15959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:45.961745 15959 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:45.963219 15959 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:45.963289 15959 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.963376 15959 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:45.963418 15959 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:45.993963 15959 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:38427
I20251027 14:54:45.993996 16086 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:38427 every 8 connection(s)
I20251027 14:54:45.995424 15959 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:54:46.003041 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 15959
Oct 27 14:54:46 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576885, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:54:46.014103 15826 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:45.998150 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:60525 (local address 127.0.104.62:39727)
1027 14:54:45.998351 (+ 201us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:45.998357 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:45.999483 (+ 1126us) server_negotiation.cc:408] Connection header received
1027 14:54:46.001041 (+ 1558us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.001046 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.001121 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.001205 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:46.002808 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.003702 (+ 894us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.005172 (+ 1470us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.005688 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.008416 (+ 2728us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:46.008449 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:46.008452 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:46.008481 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:46.011132 (+ 2651us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.011776 (+ 644us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.011780 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.011782 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.011860 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.012220 (+ 360us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.012224 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.012226 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.012558 (+ 332us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:46.012732 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.013455 (+ 723us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.013720 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:54:46.014816 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:45.998642 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:60525)
1027 14:54:45.999322 (+ 680us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:45.999360 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:46.000777 (+ 1417us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:46.001400 (+ 623us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:46.001409 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:46.001977 (+ 568us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:46.002648 (+ 671us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.002673 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.003877 (+ 1204us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.003882 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:46.005044 (+ 1162us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.005056 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.005300 (+ 244us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.006094 (+ 794us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:46.006118 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:46.008204 (+ 2086us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:46.011259 (+ 3055us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:46.011268 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:46.011286 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:46.011661 (+ 375us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:46.011958 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:46.011963 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:46.011966 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:46.012110 (+ 144us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:46.012671 (+ 561us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:46.012678 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:46.012985 (+ 307us) client_negotiation.cc:769] Sending connection context
1027 14:54:46.013342 (+ 357us) client_negotiation.cc:240] Negotiation successful
1027 14:54:46.013699 (+ 357us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":305,"threads_started":1}
I20251027 14:54:46.016314 16087 heartbeater.cc:344] Connected to a master server at 127.0.104.62:39727
I20251027 14:54:46.016775 16087 heartbeater.cc:461] Registering TS with master...
I20251027 14:54:46.017082 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.003907 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42714 (local address 127.0.104.62:39727)
1027 14:54:46.004319 (+ 412us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.004324 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.004813 (+ 489us) server_negotiation.cc:408] Connection header received
1027 14:54:46.004988 (+ 175us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.004992 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.005060 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.005152 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:46.007131 (+ 1979us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.008111 (+ 980us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.009478 (+ 1367us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.009966 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.011542 (+ 1576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:46.011582 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:46.011585 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:46.011612 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:46.014559 (+ 2947us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.015231 (+ 672us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.015235 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.015237 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.015305 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.015637 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.015641 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.015643 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.015966 (+ 323us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:46.016211 (+ 245us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.016585 (+ 374us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.016755 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":310,"thread_start_us":211,"threads_started":1}
I20251027 14:54:46.017731 16087 heartbeater.cc:507] Master 127.0.104.62:39727 requested a full tablet report, sending...
I20251027 14:54:46.019373 15638 ts_manager.cc:194] Registered new tserver with Master: 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427)
I20251027 14:54:46.020265 15638 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:60525
I20251027 14:54:46.031774 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:54:46.038200 16091 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.032653 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42720 (local address 127.0.104.62:39727)
1027 14:54:46.032795 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.032800 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.033192 (+ 392us) server_negotiation.cc:408] Connection header received
1027 14:54:46.033383 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.033388 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.033518 (+ 130us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.033659 (+ 141us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:54:46.034258 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.035133 (+ 875us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.036330 (+ 1197us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.036756 (+ 426us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.036912 (+ 156us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:54:46.037595 (+ 683us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:54:46.037741 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.037851 (+ 110us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.037950 (+ 99us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251027 14:54:46.038952 15638 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin'} at 127.0.0.1:42720:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:54:46.043936 15638 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:54:46.078833 16097 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.055150 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54796)
1027 14:54:46.070286 (+ 15136us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.070334 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:46.070535 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:46.073274 (+ 2739us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:46.073282 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:46.073313 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:46.074199 (+ 886us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.074213 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.076309 (+ 2096us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.076315 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:46.077836 (+ 1521us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.077848 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.078035 (+ 187us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.078106 (+ 71us) client_negotiation.cc:769] Sending connection context
1027 14:54:46.078277 (+ 171us) client_negotiation.cc:240] Negotiation successful
1027 14:54:46.078466 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14862,"thread_start_us":15577,"threads_started":1}
I20251027 14:54:46.079466 16102 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.055426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33565 (local address 127.0.0.1:54504)
1027 14:54:46.072684 (+ 17258us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.072702 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:46.072833 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:46.074065 (+ 1232us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:46.074068 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:46.074089 (+ 21us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:46.074805 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.074818 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.076896 (+ 2078us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.076901 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:46.078779 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.078795 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.078948 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.078971 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:54:46.079033 (+ 62us) client_negotiation.cc:240] Negotiation successful
1027 14:54:46.079152 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17161,"thread_start_us":17619,"threads_started":1}
I20251027 14:54:46.079797 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.055736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54796 (local address 127.0.104.1:32845)
1027 14:54:46.072273 (+ 16537us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.072281 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.072314 (+ 33us) server_negotiation.cc:408] Connection header received
1027 14:54:46.072451 (+ 137us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.072459 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.072754 (+ 295us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.072985 (+ 231us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:46.074367 (+ 1382us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.076131 (+ 1764us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.077979 (+ 1848us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.079060 (+ 1081us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.079216 (+ 156us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.079344 (+ 128us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.079522 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":16406,"thread_start_us":16315,"threads_started":1}
I20251027 14:54:46.081017 16101 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.056045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:38427 (local address 127.0.0.1:55668)
1027 14:54:46.073628 (+ 17583us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.073646 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:46.073769 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:46.074656 (+ 887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:46.074660 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:46.074678 (+ 18us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:46.075158 (+ 480us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.075166 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.077064 (+ 1898us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.077072 (+ 8us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:46.078798 (+ 1726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.078808 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.080553 (+ 1745us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.080577 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:54:46.080637 (+ 60us) client_negotiation.cc:240] Negotiation successful
1027 14:54:46.080751 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":17473,"spinlock_wait_cycles":15744,"thread_start_us":16378,"threads_started":1}
I20251027 14:54:46.081136 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.056256 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54504 (local address 127.0.104.2:33565)
1027 14:54:46.056911 (+ 655us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.056918 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.072849 (+ 15931us) server_negotiation.cc:408] Connection header received
1027 14:54:46.073505 (+ 656us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.073513 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.073756 (+ 243us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.074963 (+ 1207us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:46.075196 (+ 233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.076712 (+ 1516us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.079327 (+ 2615us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.080444 (+ 1117us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.080598 (+ 154us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.080711 (+ 113us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.080880 (+ 169us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":434,"thread_start_us":334,"threads_started":1}
I20251027 14:54:46.082901 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.056350 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55668 (local address 127.0.104.3:38427)
1027 14:54:46.071900 (+ 15550us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.071910 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.073910 (+ 2000us) server_negotiation.cc:408] Connection header received
1027 14:54:46.074027 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.074035 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.074246 (+ 211us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.074419 (+ 173us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:46.075325 (+ 906us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.076837 (+ 1512us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.080524 (+ 3687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.082239 (+ 1715us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.082360 (+ 121us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.082485 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.082652 (+ 167us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15388,"thread_start_us":15214,"threads_started":1}
I20251027 14:54:46.084537 15759 tablet_service.cc:1505] Processing CreateTablet for tablet c8e4cda6c3204691b53408d6bfdf39ca (DEFAULT_TABLE table=test-table [id=950d26ceb90c4d9ba6a71fcbdd1a0745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:46.084654 15891 tablet_service.cc:1505] Processing CreateTablet for tablet c8e4cda6c3204691b53408d6bfdf39ca (DEFAULT_TABLE table=test-table [id=950d26ceb90c4d9ba6a71fcbdd1a0745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:46.085861 15759 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e4cda6c3204691b53408d6bfdf39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:46.086086 15891 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e4cda6c3204691b53408d6bfdf39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:46.087067 16022 tablet_service.cc:1505] Processing CreateTablet for tablet c8e4cda6c3204691b53408d6bfdf39ca (DEFAULT_TABLE table=test-table [id=950d26ceb90c4d9ba6a71fcbdd1a0745]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:54:46.088882 16022 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c8e4cda6c3204691b53408d6bfdf39ca. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:54:46.097523 16103 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Bootstrap starting.
I20251027 14:54:46.099972 16104 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Bootstrap starting.
I20251027 14:54:46.101107 16103 tablet_bootstrap.cc:654] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:46.102440 16103 log.cc:826] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:46.103029 16105 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Bootstrap starting.
I20251027 14:54:46.103451 16104 tablet_bootstrap.cc:654] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:46.104825 16104 log.cc:826] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:46.106316 16105 tablet_bootstrap.cc:654] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Neither blocks nor log segments found. Creating new log.
I20251027 14:54:46.107519 16105 log.cc:826] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:46.112243 16104 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: No bootstrap required, opened a new log
I20251027 14:54:46.112242 16105 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: No bootstrap required, opened a new log
I20251027 14:54:46.112262 16103 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: No bootstrap required, opened a new log
I20251027 14:54:46.112552 16104 ts_tablet_manager.cc:1403] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Time spent bootstrapping tablet: real 0.013s user 0.006s sys 0.004s
I20251027 14:54:46.112567 16103 ts_tablet_manager.cc:1403] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Time spent bootstrapping tablet: real 0.015s user 0.006s sys 0.008s
I20251027 14:54:46.112660 16105 ts_tablet_manager.cc:1403] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251027 14:54:46.119693 16103 raft_consensus.cc:359] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.119663 16104 raft_consensus.cc:359] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.119663 16105 raft_consensus.cc:359] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.120054 16104 raft_consensus.cc:385] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:46.120054 16105 raft_consensus.cc:385] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:46.120059 16103 raft_consensus.cc:385] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:54:46.120152 16103 raft_consensus.cc:740] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd96d643611465cb7863e4035ce6f22, State: Initialized, Role: FOLLOWER
I20251027 14:54:46.120155 16105 raft_consensus.cc:740] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 049bc033049244cc9653fcebb6af78d9, State: Initialized, Role: FOLLOWER
I20251027 14:54:46.120157 16104 raft_consensus.cc:740] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e75be72be38a4c32a696bb85ba02f75e, State: Initialized, Role: FOLLOWER
I20251027 14:54:46.120857 16103 consensus_queue.cc:260] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.120915 16104 consensus_queue.cc:260] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.120911 16105 consensus_queue.cc:260] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.127844 15824 heartbeater.cc:499] Master 127.0.104.62:39727 was elected leader, sending a full tablet report...
I20251027 14:54:46.127904 15956 heartbeater.cc:499] Master 127.0.104.62:39727 was elected leader, sending a full tablet report...
I20251027 14:54:46.128844 16104 ts_tablet_manager.cc:1434] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Time spent starting tablet: real 0.016s user 0.007s sys 0.008s
I20251027 14:54:46.128897 16103 ts_tablet_manager.cc:1434] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Time spent starting tablet: real 0.016s user 0.007s sys 0.009s
I20251027 14:54:46.129438 16087 heartbeater.cc:499] Master 127.0.104.62:39727 was elected leader, sending a full tablet report...
I20251027 14:54:46.130091 16105 ts_tablet_manager.cc:1434] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Time spent starting tablet: real 0.017s user 0.010s sys 0.004s
W20251027 14:54:46.247020 16088 tablet.cc:2378] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:54:46.304612 15957 tablet.cc:2378] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:54:46.341442 16110 raft_consensus.cc:493] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:46.341696 16110 raft_consensus.cc:515] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.343469 16110 leader_election.cc:290] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers edd96d643611465cb7863e4035ce6f22 (127.0.104.1:32845), e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565)
I20251027 14:54:46.348824 16089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.343701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.104.3:34321)
1027 14:54:46.343917 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.343937 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:46.344080 (+ 143us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:46.344505 (+ 425us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:46.344510 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:46.344535 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:46.344978 (+ 443us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.344990 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.346457 (+ 1467us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.346461 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:46.348093 (+ 1632us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.348108 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.348249 (+ 141us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.348279 (+ 30us) client_negotiation.cc:769] Sending connection context
1027 14:54:46.348346 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:54:46.348504 (+ 158us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":105}
I20251027 14:54:46.349114 16112 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.343738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33565 (local address 127.0.104.3:38531)
1027 14:54:46.344287 (+ 549us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.344302 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:46.344443 (+ 141us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:46.344728 (+ 285us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:46.344732 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:46.344755 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:54:46.345364 (+ 609us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.345373 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.346855 (+ 1482us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.346861 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:46.348480 (+ 1619us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:46.348493 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.348659 (+ 166us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.348683 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:54:46.348743 (+ 60us) client_negotiation.cc:240] Negotiation successful
1027 14:54:46.348861 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":458,"thread_start_us":270,"threads_started":1}
I20251027 14:54:46.349730 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.343875 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:34321 (local address 127.0.104.1:32845)
1027 14:54:46.344063 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.344070 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.344092 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:54:46.344274 (+ 182us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.344280 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.344358 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.344474 (+ 116us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:46.345128 (+ 654us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.346331 (+ 1203us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.348240 (+ 1909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.349256 (+ 1016us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.349307 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.349381 (+ 74us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.349506 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":80}
I20251027 14:54:46.350118 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.344240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:38531 (local address 127.0.104.2:33565)
1027 14:54:46.344442 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.344448 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.344466 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:46.344546 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.344551 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.344618 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.344686 (+ 68us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:54:46.345503 (+ 817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.346681 (+ 1178us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.348631 (+ 1950us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.349665 (+ 1034us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.349712 (+ 47us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.349784 (+ 72us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.349910 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:54:46.350672 15779 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e4cda6c3204691b53408d6bfdf39ca" candidate_uuid: "049bc033049244cc9653fcebb6af78d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd96d643611465cb7863e4035ce6f22" is_pre_election: true
I20251027 14:54:46.350850 15911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e4cda6c3204691b53408d6bfdf39ca" candidate_uuid: "049bc033049244cc9653fcebb6af78d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75be72be38a4c32a696bb85ba02f75e" is_pre_election: true
I20251027 14:54:46.351109 15779 raft_consensus.cc:2468] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 049bc033049244cc9653fcebb6af78d9 in term 0.
I20251027 14:54:46.351272 15911 raft_consensus.cc:2468] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 049bc033049244cc9653fcebb6af78d9 in term 0.
I20251027 14:54:46.351760 15976 leader_election.cc:304] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049bc033049244cc9653fcebb6af78d9, edd96d643611465cb7863e4035ce6f22; no voters:
I20251027 14:54:46.352169 16110 raft_consensus.cc:2804] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:54:46.352303 16110 raft_consensus.cc:493] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:54:46.352422 16110 raft_consensus.cc:3060] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:46.354666 16110 raft_consensus.cc:515] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:46.355203 16110 leader_election.cc:290] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [CANDIDATE]: Term 1 election: Requested vote from peers edd96d643611465cb7863e4035ce6f22 (127.0.104.1:32845), e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565)
I20251027 14:54:46.355611 15779 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e4cda6c3204691b53408d6bfdf39ca" candidate_uuid: "049bc033049244cc9653fcebb6af78d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "edd96d643611465cb7863e4035ce6f22"
I20251027 14:54:46.355636 15911 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "c8e4cda6c3204691b53408d6bfdf39ca" candidate_uuid: "049bc033049244cc9653fcebb6af78d9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e75be72be38a4c32a696bb85ba02f75e"
I20251027 14:54:46.355777 15779 raft_consensus.cc:3060] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:46.355808 15911 raft_consensus.cc:3060] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:54:46.358079 15911 raft_consensus.cc:2468] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 049bc033049244cc9653fcebb6af78d9 in term 1.
I20251027 14:54:46.358086 15779 raft_consensus.cc:2468] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 049bc033049244cc9653fcebb6af78d9 in term 1.
I20251027 14:54:46.358469 15978 leader_election.cc:304] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 049bc033049244cc9653fcebb6af78d9, e75be72be38a4c32a696bb85ba02f75e; no voters:
I20251027 14:54:46.358742 16110 raft_consensus.cc:2804] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:54:46.359133 16110 raft_consensus.cc:697] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 1 LEADER]: Becoming Leader. State: Replica: 049bc033049244cc9653fcebb6af78d9, State: Running, Role: LEADER
I20251027 14:54:46.359601 16110 consensus_queue.cc:237] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
W20251027 14:54:46.362087 15825 tablet.cc:2378] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:54:46.364195 15638 catalog_manager.cc:5649] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 049bc033049244cc9653fcebb6af78d9 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "049bc033049244cc9653fcebb6af78d9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } health_report { overall_health: HEALTHY } } }
I20251027 14:54:46.435420 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.430565 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55670 (local address 127.0.104.3:38427)
1027 14:54:46.430729 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.430733 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.430761 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:54:46.430903 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.430908 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.430988 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.431102 (+ 114us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:54:46.431639 (+ 537us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.432608 (+ 969us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.433824 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.434182 (+ 358us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.434336 (+ 154us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:54:46.434887 (+ 551us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:54:46.434996 (+ 109us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.435071 (+ 75us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.435152 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:54:46.442785 15779 raft_consensus.cc:1275] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Refusing update from remote peer 049bc033049244cc9653fcebb6af78d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:46.442781 15911 raft_consensus.cc:1275] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Refusing update from remote peer 049bc033049244cc9653fcebb6af78d9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:54:46.443666 16113 consensus_queue.cc:1048] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:46.443888 16110 consensus_queue.cc:1048] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:54:46.454699 16117 mvcc.cc:204] Tried to move back new op lower bound from 7215418926860374016 to 7215418926533070848. Current Snapshot: MvccSnapshot[applied={T|T < 7215418926860374016}]
I20251027 14:54:46.468634 15639 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin'} at 127.0.0.1:42720:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:54:46.468871 15639 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin'} at 127.0.0.1:42720:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:54:46.472450 15639 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Sending DeleteTablet for 3 replicas of tablet c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:46.473739 15891 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e4cda6c3204691b53408d6bfdf39ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:46 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:54504
I20251027 14:54:46.473763 15759 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e4cda6c3204691b53408d6bfdf39ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:46 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:54796
I20251027 14:54:46.474020 16022 tablet_service.cc:1552] Processing DeleteTablet for tablet c8e4cda6c3204691b53408d6bfdf39ca with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:54:46 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:55668
I20251027 14:54:46.474779 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15676
I20251027 14:54:46.485915 16128 tablet_replica.cc:333] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: stopping tablet replica
I20251027 14:54:46.486167 16126 tablet_replica.cc:333] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: stopping tablet replica
I20251027 14:54:46.486403 16128 raft_consensus.cc:2243] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:54:46.486518 16126 raft_consensus.cc:2243] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:54:46.487046 16126 raft_consensus.cc:2272] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:46.487093 16128 raft_consensus.cc:2272] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:46.489106 16126 ts_tablet_manager.cc:1916] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:46.489636 16128 ts_tablet_manager.cc:1916] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:46.492890 16127 tablet_replica.cc:333] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: stopping tablet replica
I20251027 14:54:46.493414 16127 raft_consensus.cc:2243] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:54:46.494875 16126 ts_tablet_manager.cc:1929] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:46.495012 16126 log.cc:1199] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:46.495497 16126 ts_tablet_manager.cc:1950] T c8e4cda6c3204691b53408d6bfdf39ca P e75be72be38a4c32a696bb85ba02f75e: Deleting consensus metadata
I20251027 14:54:46.495525 16127 raft_consensus.cc:2272] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:54:46.497490 15626 catalog_manager.cc:4985] TS e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): tablet c8e4cda6c3204691b53408d6bfdf39ca (table test-table [id=950d26ceb90c4d9ba6a71fcbdd1a0745]) successfully deleted
I20251027 14:54:46.498436 16098 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.475631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54810 (local address 127.0.104.1:32845)
1027 14:54:46.476030 (+ 399us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.476037 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.476055 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:54:46.476118 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.476125 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.476209 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.476300 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:46.478568 (+ 2268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.480062 (+ 1494us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.482386 (+ 2324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.482915 (+ 529us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.485882 (+ 2967us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:46.485932 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:46.485952 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:46.485992 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:46.494639 (+ 8647us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.495694 (+ 1055us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.495705 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.495709 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.495816 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.496314 (+ 498us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.496319 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.496322 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.496728 (+ 406us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:46.497097 (+ 369us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.497848 (+ 751us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.498097 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":264}
I20251027 14:54:46.499300 15819 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:46.499326 16127 ts_tablet_manager.cc:1916] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:54:46.499866 16128 ts_tablet_manager.cc:1929] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:54:46.500080 16128 log.cc:1199] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:46.500792 16128 ts_tablet_manager.cc:1950] T c8e4cda6c3204691b53408d6bfdf39ca P 049bc033049244cc9653fcebb6af78d9: Deleting consensus metadata
I20251027 14:54:46.503033 15623 catalog_manager.cc:4985] TS 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427): tablet c8e4cda6c3204691b53408d6bfdf39ca (table test-table [id=950d26ceb90c4d9ba6a71fcbdd1a0745]) successfully deleted
I20251027 14:54:46.598901 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15676
W20251027 14:54:46.614692 15624 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:32845 (error 108)
I20251027 14:54:46.614964 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15828
I20251027 14:54:46.616768 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.616088 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54814)
1027 14:54:46.616507 (+ 419us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.616592 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":326,"thread_start_us":197,"threads_started":1}
W20251027 14:54:46.617059 15624 catalog_manager.cc:4712] TS edd96d643611465cb7863e4035ce6f22 (127.0.104.1:32845): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet c8e4cda6c3204691b53408d6bfdf39ca: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
I20251027 14:54:46.630286 16099 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.615821 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54508 (local address 127.0.104.2:33565)
1027 14:54:46.616012 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.616019 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.616041 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:54:46.616111 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.616118 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.616199 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.616301 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:46.618513 (+ 2212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.619895 (+ 1382us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.621399 (+ 1504us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.621885 (+ 486us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.623567 (+ 1682us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:46.623615 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:46.623636 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:46.623675 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:46.627474 (+ 3799us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.628103 (+ 629us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.628113 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.628117 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.628211 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.628623 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.628628 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.628632 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.629036 (+ 404us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:46.629362 (+ 326us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.629663 (+ 301us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.629924 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:54:46.630957 15951 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:46.670473 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.669959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54818)
1027 14:54:46.670190 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.670296 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":83}
I20251027 14:54:46.671691 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.671279 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54834)
1027 14:54:46.671421 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.671517 (+ 96us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20251027 14:54:46.721884 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15828
I20251027 14:54:46.732336 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.731804 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54838)
1027 14:54:46.732047 (+ 243us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.732160 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":88}
I20251027 14:54:46.733577 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.733168 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54840)
1027 14:54:46.733312 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.733405 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":48}
I20251027 14:54:46.737721 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15959
I20251027 14:54:46.752642 16100 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.738404 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55678 (local address 127.0.104.3:38427)
1027 14:54:46.738631 (+ 227us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:46.738638 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:46.738670 (+ 32us) server_negotiation.cc:408] Connection header received
1027 14:54:46.738751 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:46.738757 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:46.738846 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:46.738959 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:46.740451 (+ 1492us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.741856 (+ 1405us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:46.743192 (+ 1336us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:46.743659 (+ 467us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:46.745505 (+ 1846us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:46.745557 (+ 52us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:46.745576 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:46.745614 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:46.749794 (+ 4180us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.750467 (+ 673us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.750478 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.750481 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.750608 (+ 127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:46.750912 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:46.750917 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:46.750920 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:46.751330 (+ 410us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:46.751649 (+ 319us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:46.751988 (+ 339us) server_negotiation.cc:300] Negotiation successful
1027 14:54:46.752252 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":100}
I20251027 14:54:46.753404 16082 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:46.814946 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.814382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54846)
1027 14:54:46.814630 (+ 248us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.814759 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20251027 14:54:46.816051 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.815617 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54850)
1027 14:54:46.815812 (+ 195us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.815890 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20251027 14:54:46.871086 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15959
I20251027 14:54:46.887102 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 15606
I20251027 14:54:46.888258 15669 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:54:46.998546 16131 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:46.997962 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:32845 (local address 127.0.0.1:54866)
1027 14:54:46.998239 (+ 277us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:46.998341 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:32845: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":160}
I20251027 14:54:46.998528 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 15606
I20251027 14:54:47.016268 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39727
--webserver_interface=127.0.104.62
--webserver_port=36661
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:39727
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:47.338928 16135 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:47.339262 16135 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:47.339401 16135 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:47.347389 16135 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:54:47.347499 16135 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:47.347579 16135 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:47.347642 16135 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:54:47.347733 16135 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:54:47.358153 16135 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:39727
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:39727
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=36661
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:47.359730 16135 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:47.361090 16135 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:47.368628 16141 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:47.368891 16143 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:47.368635 16140 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:47.369750 16135 server_base.cc:1047] running on GCE node
I20251027 14:54:47.370422 16135 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:47.371706 16135 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:47.372910 16135 hybrid_clock.cc:648] HybridClock initialized: now 1761576887372847 us; error 40 us; skew 500 ppm
I20251027 14:54:47.373209 16135 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:47 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:47.376257 16135 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:54:47.377785 16135 webserver.cc:492] Webserver started at http://127.0.104.62:36661/ using document root <none> and password file <none>
I20251027 14:54:47.378479 16135 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:47.378562 16135 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:47.383798 16135 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:54:47.386518 16150 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:47.387729 16135 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:54:47.387884 16135 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "e625969dad8747ffb083fc47c71eb5bd"
format_stamp: "Formatted at 2025-10-27 14:54:41 on dist-test-slave-hxxt"
server_key: "01a4d8b63c49655046550a6011f8b748"
server_key_iv: "aab154fc1c62670a4107b7abdbd61c94"
server_key_version: "encryptionkey@0"
I20251027 14:54:47.388629 16135 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:47.418056 16135 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:47.422080 16135 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:47.422345 16135 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:47.440426 16135 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:39727
I20251027 14:54:47.440466 16201 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:39727 every 8 connection(s)
I20251027 14:54:47.441987 16135 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:54:47.446713 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16135
I20251027 14:54:47.447528 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:32845
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:39727
--webserver_port=38519
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:47.449395 16202 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:54:47.452623 16202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap starting.
I20251027 14:54:47.477242 16202 log.cc:826] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:47.497269 16202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=7 ignored=0} mutations{seen=9 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:54:47.497758 16202 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd: Bootstrap complete.
I20251027 14:54:47.504256 16202 raft_consensus.cc:359] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:47.504935 16202 raft_consensus.cc:740] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e625969dad8747ffb083fc47c71eb5bd, State: Initialized, Role: FOLLOWER
I20251027 14:54:47.505525 16202 consensus_queue.cc:260] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:47.505729 16202 raft_consensus.cc:399] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:54:47.505844 16202 raft_consensus.cc:493] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:54:47.505990 16202 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 2 FOLLOWER]: Advancing to term 3
I20251027 14:54:47.509539 16202 raft_consensus.cc:515] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:47.509999 16202 leader_election.cc:304] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e625969dad8747ffb083fc47c71eb5bd; no voters:
I20251027 14:54:47.510578 16202 leader_election.cc:290] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [CANDIDATE]: Term 3 election: Requested vote from peers
I20251027 14:54:47.510761 16206 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 3 FOLLOWER]: Leader election won for term 3
I20251027 14:54:47.512528 16202 sys_catalog.cc:565] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:54:47.517205 16206 raft_consensus.cc:697] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [term 3 LEADER]: Becoming Leader. State: Replica: e625969dad8747ffb083fc47c71eb5bd, State: Running, Role: LEADER
I20251027 14:54:47.517719 16206 consensus_queue.cc:237] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } }
I20251027 14:54:47.520167 16207 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "e625969dad8747ffb083fc47c71eb5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } } }
I20251027 14:54:47.520491 16207 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:47.520586 16208 sys_catalog.cc:455] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader e625969dad8747ffb083fc47c71eb5bd. Latest consensus state: current_term: 3 leader_uuid: "e625969dad8747ffb083fc47c71eb5bd" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e625969dad8747ffb083fc47c71eb5bd" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 39727 } } }
I20251027 14:54:47.520879 16208 sys_catalog.cc:458] T 00000000000000000000000000000000 P e625969dad8747ffb083fc47c71eb5bd [sys.catalog]: This master's current role is: LEADER
I20251027 14:54:47.521641 16221 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:54:47.534844 16221 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:54:47.536875 16221 catalog_manager.cc:1269] Loaded cluster ID: d31fc0346fd84120b95b882286b094d6
I20251027 14:54:47.536974 16221 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
F20251027 14:54:47.538800 16221 catalog_manager.cc:1476] Initializing Kudu internal certificate authority failed: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fcfb8a69dcd google::LogMessage::Fail() at ??:0
@ 0x7fcfb8a6db93 google::LogMessage::SendToLog() at ??:0
@ 0x7fcfb8a697cc google::LogMessage::Flush() at ??:0
@ 0x7fcfb8a6af59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fcfcf2ad7df kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fcfcf2aab86 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fcfcf33b9c0 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fcfcf33b7e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7fcfbb7fed80 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fcfbb80204c kudu::ThreadPool::CreateThread()::$_1::operator()() at ??:0
@ 0x7fcfbb801ec9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7fcfbb7ddee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fcfbede56db start_thread at ??:0
@ 0x7fcfb791a71f clone at ??:0
W20251027 14:54:47.852339 16204 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:47.852670 16204 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:47.852767 16204 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:47.861094 16204 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:47.861208 16204 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:47.861368 16204 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:54:47.871910 16204 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:32845
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=38519
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:47.873499 16204 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:47.874809 16204 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:47.884116 16244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:47.884107 16243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:47.884722 16246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:47.885098 16204 server_base.cc:1047] running on GCE node
I20251027 14:54:47.885730 16204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:47.886628 16204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:47.887835 16204 hybrid_clock.cc:648] HybridClock initialized: now 1761576887887768 us; error 49 us; skew 500 ppm
I20251027 14:54:47.888099 16204 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:47 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:47.891207 16204 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:54:47.892697 16204 webserver.cc:492] Webserver started at http://127.0.104.1:38519/ using document root <none> and password file <none>
I20251027 14:54:47.893452 16204 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:47.893540 16204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:47.899150 16204 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20251027 14:54:47.902278 16253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:47.903685 16204 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:54:47.903852 16204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "edd96d643611465cb7863e4035ce6f22"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "1702941f30bec64bea5af661fbbb15e8"
server_key_iv: "da36bae8087451443a40921ad0069864"
server_key_version: "encryptionkey@0"
I20251027 14:54:47.904589 16204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:47.922407 16204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:47.926218 16204 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:47.926501 16204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:47.927634 16204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:47.929731 16260 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20251027 14:54:47.935586 16204 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251027 14:54:47.935691 16204 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.006s user 0.001s sys 0.000s
I20251027 14:54:47.935822 16204 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251027 14:54:47.938850 16204 ts_tablet_manager.cc:616] Registered 1 tablets
I20251027 14:54:47.938947 16204 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s user 0.001s sys 0.000s
I20251027 14:54:47.939287 16260 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Bootstrap starting.
I20251027 14:54:47.947137 16260 log.cc:826] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Log is configured to *not* fsync() on all Append() calls
I20251027 14:54:47.964334 16260 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:54:47.965073 16260 tablet_bootstrap.cc:492] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Bootstrap complete.
I20251027 14:54:47.965654 16260 ts_tablet_manager.cc:1403] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Time spent bootstrapping tablet: real 0.027s user 0.016s sys 0.009s
I20251027 14:54:47.972512 16204 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:32845
I20251027 14:54:47.972561 16367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:32845 every 8 connection(s)
I20251027 14:54:47.974171 16204 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:54:47.973510 16260 raft_consensus.cc:359] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:47.974409 16260 raft_consensus.cc:740] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: edd96d643611465cb7863e4035ce6f22, State: Initialized, Role: FOLLOWER
I20251027 14:54:47.975324 16260 consensus_queue.cc:260] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:47.977149 16260 ts_tablet_manager.cc:1434] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22: Time spent starting tablet: real 0.011s user 0.011s sys 0.001s
I20251027 14:54:47.979142 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:47.977435 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:59281)
1027 14:54:47.978125 (+ 690us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:47.978298 (+ 173us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":279,"threads_started":1}
W20251027 14:54:47.979646 16368 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:39727 (0 consecutive failures): Network error: Failed to ping master at 127.0.104.62:39727: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
I20251027 14:54:47.980404 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:47.980092 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:33179)
1027 14:54:47.980206 (+ 114us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:47.980277 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
I20251027 14:54:47.981153 16371 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:47.980888 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:39103)
1027 14:54:47.980995 (+ 107us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:47.981048 (+ 53us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":37}
W20251027 14:54:47.981335 16368 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251027 14:54:47.984653 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16204
I20251027 14:54:47.985251 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:33565
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:39727
--webserver_port=37511
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:54:48.321003 16372 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:48.321298 16372 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:48.321405 16372 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:48.329428 16372 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:48.329533 16372 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:48.329684 16372 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:54:48.339844 16372 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:33565
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=37511
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:48.341369 16372 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:48.342671 16372 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:48.351476 16377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:48.352059 16380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:48.352077 16378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:48.353093 16372 server_base.cc:1047] running on GCE node
I20251027 14:54:48.353674 16372 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:48.354763 16372 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:48.355991 16372 hybrid_clock.cc:648] HybridClock initialized: now 1761576888355931 us; error 40 us; skew 500 ppm
I20251027 14:54:48.356274 16372 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:48 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:48.359334 16372 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:54:48.360926 16372 webserver.cc:492] Webserver started at http://127.0.104.2:37511/ using document root <none> and password file <none>
I20251027 14:54:48.361636 16372 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:48.361725 16372 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:48.367074 16372 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.003s
I20251027 14:54:48.369699 16387 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:48.370913 16372 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:54:48.371054 16372 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "e75be72be38a4c32a696bb85ba02f75e"
format_stamp: "Formatted at 2025-10-27 14:54:42 on dist-test-slave-hxxt"
server_key: "86bc93cd3d25c4fc2a088eb8c5f64354"
server_key_iv: "7cbabd74cac04be7c82b4dba551afdff"
server_key_version: "encryptionkey@0"
I20251027 14:54:48.371721 16372 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:48.384027 16372 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:48.387794 16372 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:48.388044 16372 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:48.388974 16372 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:48.390427 16372 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:48.390499 16372 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:48.390619 16372 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:48.390671 16372 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:48.422262 16372 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33565
I20251027 14:54:48.422317 16499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33565 every 8 connection(s)
I20251027 14:54:48.423661 16372 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:54:48.426692 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16372
I20251027 14:54:48.427287 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:38427
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:39727
--webserver_port=43967
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40301
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:54:48.428748 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.426643 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:50539)
1027 14:54:48.427473 (+ 830us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.427688 (+ 215us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":375,"threads_started":1}
W20251027 14:54:48.429334 16500 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:39727 (0 consecutive failures): Network error: Failed to ping master at 127.0.104.62:39727: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
I20251027 14:54:48.430155 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.429836 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:39481)
1027 14:54:48.429968 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.430044 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251027 14:54:48.430994 16502 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.430663 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:50471)
1027 14:54:48.430790 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.430856 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
W20251027 14:54:48.431267 16500 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251027 14:54:48.743736 16503 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:54:48.744026 16503 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:54:48.744151 16503 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:54:48.752148 16503 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:54:48.752240 16503 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:54:48.752465 16503 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:54:48.762766 16503 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40301
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:38427
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=43967
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:39727
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:54:48.764289 16503 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:54:48.765636 16503 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:54:48.774613 16508 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:48.774646 16509 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:54:48.775209 16511 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:54:48.775871 16503 server_base.cc:1047] running on GCE node
I20251027 14:54:48.776485 16503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:54:48.777385 16503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:54:48.778592 16503 hybrid_clock.cc:648] HybridClock initialized: now 1761576888778526 us; error 36 us; skew 500 ppm
I20251027 14:54:48.778860 16503 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:54:48 dist-test-slave-hxxt krb5kdc[15045](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576888, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:54:48.781901 16503 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:54:48.783310 16503 webserver.cc:492] Webserver started at http://127.0.104.3:43967/ using document root <none> and password file <none>
I20251027 14:54:48.784016 16503 fs_manager.cc:362] Metadata directory not provided
I20251027 14:54:48.784093 16503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:54:48.789494 16503 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:54:48.792081 16518 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:48.793380 16503 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:54:48.793529 16503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "049bc033049244cc9653fcebb6af78d9"
format_stamp: "Formatted at 2025-10-27 14:54:43 on dist-test-slave-hxxt"
server_key: "3d6ba4f73b6aa6ef032bdf6e719f63f5"
server_key_iv: "93f04e55b26a40da36c848c7f3a69abe"
server_key_version: "encryptionkey@0"
I20251027 14:54:48.794173 16503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:54:48.813761 16503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:54:48.817629 16503 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:54:48.817899 16503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:54:48.818836 16503 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:54:48.820338 16503 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:54:48.820436 16503 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20251027 14:54:48.820560 16503 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:54:48.820614 16503 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:54:48.851466 16503 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:38427
I20251027 14:54:48.851502 16630 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:38427 every 8 connection(s)
I20251027 14:54:48.852916 16503 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedIPKICert.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:54:48.857761 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.855879 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:34453)
1027 14:54:48.856673 (+ 794us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.856851 (+ 178us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":335,"threads_started":1}
W20251027 14:54:48.858434 16631 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:39727 (0 consecutive failures): Network error: Failed to ping master at 127.0.104.62:39727: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
I20251027 14:54:48.858726 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16503
I20251027 14:54:48.859292 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.858902 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:38557)
1027 14:54:48.859055 (+ 153us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.859148 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251027 14:54:48.860071 16633 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.859775 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:49563)
1027 14:54:48.859900 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.859960 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:54:48.860304 16631 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251027 14:54:48.982452 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:48.981793 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:36387)
1027 14:54:48.982169 (+ 376us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:48.982240 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":295,"thread_start_us":196,"threads_started":1}
I20251027 14:54:49.303942 16636 raft_consensus.cc:493] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:49.304205 16636 raft_consensus.cc:515] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:49.305819 16636 leader_election.cc:290] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565), 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427)
Oct 27 14:54:49 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
Oct 27 14:54:49 dist-test-slave-hxxt krb5kdc[15045](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576887, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:54:49.329118 16639 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.306058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:38427 (local address 127.0.104.1:36363)
1027 14:54:49.308281 (+ 2223us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:49.308299 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:49.308444 (+ 145us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:49.310857 (+ 2413us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:49.310864 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:49.311448 (+ 584us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:49.311999 (+ 551us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:49.312011 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.313565 (+ 1554us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:49.313571 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:49.314465 (+ 894us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:49.314477 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.314693 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:49.319784 (+ 5091us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:49.319820 (+ 36us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:49.322668 (+ 2848us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:49.326081 (+ 3413us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:49.326085 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:49.326087 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:49.326447 (+ 360us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:49.326797 (+ 350us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:49.326802 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:49.326805 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:49.326935 (+ 130us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:49.327623 (+ 688us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:49.327630 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:49.327871 (+ 241us) client_negotiation.cc:769] Sending connection context
1027 14:54:49.328193 (+ 322us) client_negotiation.cc:240] Negotiation successful
1027 14:54:49.328682 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2126,"mutex_wait_us":183,"spinlock_wait_cycles":28800,"thread_start_us":1797,"threads_started":1}
I20251027 14:54:49.329216 16638 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.306352 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:36363 (local address 127.0.104.3:38427)
1027 14:54:49.308090 (+ 1738us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:49.308104 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:49.308419 (+ 315us) server_negotiation.cc:408] Connection header received
1027 14:54:49.309607 (+ 1188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:49.309620 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:49.310013 (+ 393us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:49.310395 (+ 382us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:49.312165 (+ 1770us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.313408 (+ 1243us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:49.314601 (+ 1193us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.315101 (+ 500us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:49.322847 (+ 7746us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:49.322922 (+ 75us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:49.322936 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:49.322970 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:49.325938 (+ 2968us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:49.326578 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:49.326585 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:49.326592 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:49.326693 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:49.327033 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:49.327036 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:49.327038 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:49.327473 (+ 435us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:49.327644 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:49.328442 (+ 798us) server_negotiation.cc:300] Negotiation successful
1027 14:54:49.328825 (+ 383us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1575,"thread_start_us":1448,"threads_started":1}
I20251027 14:54:49.329203 16635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.306045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33565 (local address 127.0.104.1:49805)
1027 14:54:49.306279 (+ 234us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:49.306319 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1027 14:54:49.306982 (+ 663us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:54:49.310185 (+ 3203us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:54:49.310197 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:54:49.311103 (+ 906us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:54:49.311968 (+ 865us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:49.311990 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.313603 (+ 1613us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:54:49.313607 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:54:49.314448 (+ 841us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:54:49.314459 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.314739 (+ 280us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:49.319784 (+ 5045us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:54:49.319816 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:54:49.322380 (+ 2564us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:54:49.325925 (+ 3545us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:49.325933 (+ 8us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:49.325949 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:49.326459 (+ 510us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:49.326826 (+ 367us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:54:49.326829 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:54:49.326832 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:54:49.326925 (+ 93us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:54:49.327620 (+ 695us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:54:49.327628 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:54:49.327871 (+ 243us) client_negotiation.cc:769] Sending connection context
1027 14:54:49.328204 (+ 333us) client_negotiation.cc:240] Negotiation successful
1027 14:54:49.328828 (+ 624us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":103,"mutex_wait_us":183,"spinlock_wait_cycles":19072}
I20251027 14:54:49.329186 16637 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.306287 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:49805 (local address 127.0.104.2:33565)
1027 14:54:49.307781 (+ 1494us) server_negotiation.cc:207] Beginning negotiation
1027 14:54:49.307799 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1027 14:54:49.307838 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:54:49.309236 (+ 1398us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:54:49.309245 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:54:49.309720 (+ 475us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:54:49.310012 (+ 292us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:54:49.312157 (+ 2145us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.313445 (+ 1288us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:54:49.314583 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:54:49.315078 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:54:49.322663 (+ 7585us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:54:49.322767 (+ 104us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:54:49.322791 (+ 24us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:54:49.322836 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:54:49.325776 (+ 2940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:49.326595 (+ 819us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:49.326602 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:49.326608 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:49.326710 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:54:49.327021 (+ 311us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:54:49.327026 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:54:49.327029 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:54:49.327470 (+ 441us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:54:49.327644 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1027 14:54:49.328446 (+ 802us) server_negotiation.cc:300] Negotiation successful
1027 14:54:49.328837 (+ 391us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1321,"thread_start_us":1204,"threads_started":1}
W20251027 14:54:49.332412 16257 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
W20251027 14:54:49.332702 16254 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:49.332829 16254 leader_election.cc:304] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edd96d643611465cb7863e4035ce6f22; no voters: 049bc033049244cc9653fcebb6af78d9, e75be72be38a4c32a696bb85ba02f75e
I20251027 14:54:49.333206 16636 raft_consensus.cc:2749] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:54:49.432457 16640 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.431702 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:54391)
1027 14:54:49.432093 (+ 391us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:49.432197 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":306,"thread_start_us":190,"threads_started":1}
I20251027 14:54:49.861495 16641 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.860782 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:50851)
1027 14:54:49.861189 (+ 407us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:49.861279 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":221,"threads_started":1}
I20251027 14:54:49.983850 16642 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:49.983210 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:45373)
1027 14:54:49.983614 (+ 404us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:49.983689 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":317,"thread_start_us":234,"threads_started":1}
I20251027 14:54:50.433912 16643 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:50.433276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:44989)
1027 14:54:50.433682 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:50.433754 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":320,"thread_start_us":218,"threads_started":1}
I20251027 14:54:50.863070 16644 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:50.862356 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:48147)
1027 14:54:50.862768 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:50.862859 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":218,"threads_started":1}
I20251027 14:54:50.985544 16645 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:50.984723 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:38851)
1027 14:54:50.985266 (+ 543us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:50.985368 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":340,"threads_started":1}
I20251027 14:54:51.174120 16646 raft_consensus.cc:493] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:51.174412 16646 raft_consensus.cc:515] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:51.175143 16646 leader_election.cc:290] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565), 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427)
W20251027 14:54:51.176090 16257 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
W20251027 14:54:51.176267 16254 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:51.176416 16254 leader_election.cc:304] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edd96d643611465cb7863e4035ce6f22; no voters: 049bc033049244cc9653fcebb6af78d9, e75be72be38a4c32a696bb85ba02f75e
I20251027 14:54:51.176666 16646 raft_consensus.cc:2749] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:54:51.435413 16647 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:51.434710 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:52561)
1027 14:54:51.435136 (+ 426us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:51.435216 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":323,"thread_start_us":215,"threads_started":1}
I20251027 14:54:51.864812 16648 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:51.864013 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:55899)
1027 14:54:51.864510 (+ 497us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:51.864608 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":393,"thread_start_us":259,"threads_started":1}
I20251027 14:54:51.987147 16649 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:51.986391 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:46533)
1027 14:54:51.986816 (+ 425us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:51.986906 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":237,"threads_started":1}
I20251027 14:54:52.437145 16650 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:52.436441 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:44451)
1027 14:54:52.436864 (+ 423us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:52.436942 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":328,"thread_start_us":210,"threads_started":1}
I20251027 14:54:52.866297 16651 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:52.865601 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:60431)
1027 14:54:52.866018 (+ 417us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:52.866092 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":307,"thread_start_us":203,"threads_started":1}
I20251027 14:54:52.964484 16652 raft_consensus.cc:493] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:52.964665 16652 raft_consensus.cc:515] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:52.965329 16652 leader_election.cc:290] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565), 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427)
W20251027 14:54:52.966212 16254 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
W20251027 14:54:52.966382 16257 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:52.966521 16257 leader_election.cc:304] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edd96d643611465cb7863e4035ce6f22; no voters: 049bc033049244cc9653fcebb6af78d9, e75be72be38a4c32a696bb85ba02f75e
I20251027 14:54:52.966768 16652 raft_consensus.cc:2749] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:54:52.988965 16653 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:52.988153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:46131)
1027 14:54:52.988677 (+ 524us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:52.988776 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":436,"thread_start_us":251,"threads_started":1}
I20251027 14:54:53.438732 16654 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:53.438057 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:42717)
1027 14:54:53.438506 (+ 449us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:53.438582 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":245,"threads_started":1}
I20251027 14:54:53.867718 16655 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:53.867091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:37509)
1027 14:54:53.867488 (+ 397us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:53.867562 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":309,"thread_start_us":202,"threads_started":1}
I20251027 14:54:53.990381 16656 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:53.989737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:60571)
1027 14:54:53.990149 (+ 412us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:53.990218 (+ 69us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":222,"threads_started":1}
I20251027 14:54:54.440253 16657 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:54.439585 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:58011)
1027 14:54:54.439987 (+ 402us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:54.440064 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":214,"threads_started":1}
I20251027 14:54:54.869346 16658 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:54.868651 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:49441)
1027 14:54:54.869088 (+ 437us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:54.869171 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":337,"thread_start_us":223,"threads_started":1}
I20251027 14:54:54.991976 16659 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:54.991219 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:51469)
1027 14:54:54.991634 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:54.991750 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":300,"thread_start_us":194,"threads_started":1}
I20251027 14:54:55.441819 16660 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:55.441116 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:35955)
1027 14:54:55.441566 (+ 450us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:55.441643 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":202,"threads_started":1}
I20251027 14:54:55.870916 16661 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:55.870163 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:33143)
1027 14:54:55.870617 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:55.870695 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":365,"thread_start_us":211,"threads_started":1}
I20251027 14:54:55.993492 16662 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:55.992849 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:46479)
1027 14:54:55.993267 (+ 418us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:55.993338 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":332,"thread_start_us":207,"threads_started":1}
I20251027 14:54:56.443277 16663 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:56.442633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:41421)
1027 14:54:56.443037 (+ 404us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:56.443107 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":221,"threads_started":1}
I20251027 14:54:56.872766 16664 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:56.871966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:44275)
1027 14:54:56.872493 (+ 527us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:56.872580 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":290,"threads_started":1}
I20251027 14:54:56.995759 16665 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:56.994436 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:54411)
1027 14:54:56.995406 (+ 970us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:56.995504 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":850,"thread_start_us":356,"threads_started":1}
I20251027 14:54:57.444818 16666 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:57.444118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:45919)
1027 14:54:57.444568 (+ 450us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:57.444643 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":360,"thread_start_us":208,"threads_started":1}
I20251027 14:54:57.874398 16667 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:57.873656 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:35395)
1027 14:54:57.874150 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:57.874229 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":395,"thread_start_us":295,"threads_started":1}
I20251027 14:54:57.976089 16668 raft_consensus.cc:493] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:54:57.976312 16668 raft_consensus.cc:515] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "edd96d643611465cb7863e4035ce6f22" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 32845 } } peers { permanent_uuid: "e75be72be38a4c32a696bb85ba02f75e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33565 } } peers { permanent_uuid: "049bc033049244cc9653fcebb6af78d9" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 38427 } }
I20251027 14:54:57.977099 16668 leader_election.cc:290] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565), 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427)
W20251027 14:54:57.977972 16257 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer e75be72be38a4c32a696bb85ba02f75e (127.0.104.2:33565): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
W20251027 14:54:57.978134 16254 leader_election.cc:343] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 049bc033049244cc9653fcebb6af78d9 (127.0.104.3:38427): Not found: Tablet not found: c8e4cda6c3204691b53408d6bfdf39ca
I20251027 14:54:57.978259 16254 leader_election.cc:304] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: edd96d643611465cb7863e4035ce6f22; no voters: 049bc033049244cc9653fcebb6af78d9, e75be72be38a4c32a696bb85ba02f75e
I20251027 14:54:57.978467 16668 raft_consensus.cc:2749] T c8e4cda6c3204691b53408d6bfdf39ca P edd96d643611465cb7863e4035ce6f22 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:54:57.997604 16669 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:57.996868 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:48375)
1027 14:54:57.997316 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:57.997414 (+ 98us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20251027 14:54:58.446503 16670 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:58.445761 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:37439)
1027 14:54:58.446217 (+ 456us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:58.446298 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":219,"threads_started":1}
I20251027 14:54:58.875949 16671 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:58.875238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:42981)
1027 14:54:58.875695 (+ 457us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:58.875774 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":241,"threads_started":1}
I20251027 14:54:58.999136 16672 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:58.998447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:54705)
1027 14:54:58.998853 (+ 406us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:58.998931 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":311,"thread_start_us":207,"threads_started":1}
I20251027 14:54:59.448035 16673 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:59.447315 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:38765)
1027 14:54:59.447769 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:59.447863 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":355,"thread_start_us":233,"threads_started":1}
I20251027 14:54:59.877651 16674 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:54:59.876883 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:36137)
1027 14:54:59.877340 (+ 457us) negotiation.cc:107] Waiting for socket to connect
1027 14:54:59.877440 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":352,"thread_start_us":242,"threads_started":1}
I20251027 14:55:00.000720 16675 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:00.000010 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:40103)
1027 14:55:00.000476 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:00.000552 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":374,"thread_start_us":254,"threads_started":1}
I20251027 14:55:00.449666 16676 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:00.448921 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:42797)
1027 14:55:00.449416 (+ 495us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:00.449496 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":394,"thread_start_us":266,"threads_started":1}
I20251027 14:55:00.879271 16677 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:00.878564 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:41063)
1027 14:55:00.879012 (+ 448us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:00.879096 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":229,"threads_started":1}
I20251027 14:55:01.002404 16678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:01.001684 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:57817)
1027 14:55:01.002133 (+ 449us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:01.002222 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":222,"threads_started":1}
I20251027 14:55:01.451251 16679 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:01.450500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:44423)
1027 14:55:01.450978 (+ 478us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:01.451090 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":379,"thread_start_us":240,"threads_started":1}
I20251027 14:55:01.880903 16680 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:01.880157 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:43309)
1027 14:55:01.880616 (+ 459us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:01.880724 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":368,"thread_start_us":259,"threads_started":1}
I20251027 14:55:02.003875 16681 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:02.003207 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:55699)
1027 14:55:02.003617 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:02.003693 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":217,"threads_started":1}
I20251027 14:55:02.452895 16682 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:02.452118 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:56393)
1027 14:55:02.452622 (+ 504us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:02.452702 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":406,"thread_start_us":293,"threads_started":1}
I20251027 14:55:02.882395 16683 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:02.881713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:43443)
1027 14:55:02.882113 (+ 400us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:02.882186 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":312,"thread_start_us":216,"threads_started":1}
I20251027 14:55:03.005739 16684 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:03.005073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.1:55865)
1027 14:55:03.005487 (+ 414us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:03.005565 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":188,"threads_started":1}
I20251027 14:55:03.454687 16685 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:03.453988 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.2:58627)
1027 14:55:03.454419 (+ 431us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:03.454487 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":198,"threads_started":1}
I20251027 14:55:03.863169 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 16204
I20251027 14:55:03.882700 16686 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:03.864279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49868 (local address 127.0.104.1:32845)
1027 14:55:03.864775 (+ 496us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:03.864785 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:03.864816 (+ 31us) server_negotiation.cc:408] Connection header received
1027 14:55:03.864897 (+ 81us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:03.864907 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:03.865132 (+ 225us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:03.865324 (+ 192us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:03.867767 (+ 2443us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:03.868970 (+ 1203us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:03.870195 (+ 1225us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:03.870677 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:03.877197 (+ 6520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:03.877262 (+ 65us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:03.877278 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:03.877314 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:03.879935 (+ 2621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:03.880690 (+ 755us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:03.880698 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:03.880700 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:03.880794 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:03.881149 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:03.881153 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:03.881155 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:03.881475 (+ 320us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:03.881757 (+ 282us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:03.882078 (+ 321us) server_negotiation.cc:300] Negotiation successful
1027 14:55:03.882370 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":350,"thread_start_us":254,"threads_started":1}
I20251027 14:55:03.884128 16687 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:03.883479 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:39727 (local address 127.0.104.3:42345)
1027 14:55:03.883921 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:03.883992 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:39727: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":351,"thread_start_us":204,"threads_started":1}
I20251027 14:55:03.884121 16363 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:03.971179 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 16204
I20251027 14:55:03.985759 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 16372
I20251027 14:55:03.998234 16689 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:03.986495 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42168 (local address 127.0.104.2:33565)
1027 14:55:03.986954 (+ 459us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:03.986959 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:03.986973 (+ 14us) server_negotiation.cc:408] Connection header received
1027 14:55:03.987025 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:03.987030 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:03.987088 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:03.987191 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:03.988854 (+ 1663us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:03.989792 (+ 938us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:03.990956 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:03.991363 (+ 407us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:03.993431 (+ 2068us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:03.993470 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:03.993473 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:03.993504 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:03.995845 (+ 2341us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:03.996493 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:03.996498 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:03.996501 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:03.996583 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:03.996901 (+ 318us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:03.996904 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:03.996906 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:03.997132 (+ 226us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:03.997390 (+ 258us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:03.997664 (+ 274us) server_negotiation.cc:300] Negotiation successful
1027 14:55:03.997920 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":349,"thread_start_us":222,"threads_started":1}
I20251027 14:55:03.998991 16495 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:04.082408 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 16372
I20251027 14:55:04.097508 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 16503
I20251027 14:55:04.109624 16691 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:04.098307 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36108 (local address 127.0.104.3:38427)
1027 14:55:04.098927 (+ 620us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:04.098931 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:04.098948 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:55:04.099005 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:04.099009 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:04.099072 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:04.099171 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:04.100951 (+ 1780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:04.101935 (+ 984us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:04.103024 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:04.103353 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:04.105124 (+ 1771us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:04.105162 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:04.105166 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:04.105191 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:04.107607 (+ 2416us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:04.108184 (+ 577us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:04.108187 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:04.108190 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:04.108260 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:04.108529 (+ 269us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:04.108533 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:04.108535 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:04.108736 (+ 201us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:04.108911 (+ 175us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:04.109161 (+ 250us) server_negotiation.cc:300] Negotiation successful
1027 14:55:04.109355 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":480,"thread_start_us":360,"threads_started":1}
I20251027 14:55:04.110318 16626 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:04.193754 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 16503
2025-10-27T14:55:04Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedIPKICert (25043 ms)
[ RUN ] SecurityITest.TestEnableAndDisableEncryptedTSK
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:55:04 dist-test-slave-hxxt krb5kdc[16695](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:55:04 dist-test-slave-hxxt krb5kdc[16695](info): set up 2 sockets
Oct 27 14:55:04 dist-test-slave-hxxt krb5kdc[16695](info): commencing operation
krb5kdc: starting...
W20251027 14:55:06.080937 416 mini_kdc.cc:121] Time spent starting KDC: real 1.847s user 0.004s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:55:06 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:55:06Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:55:06Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:55:06.345923 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37463
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:06.669988 16711 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:06.670284 16711 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:06.670416 16711 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:06.678320 16711 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251027 14:55:06.678406 16711 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251027 14:55:06.678473 16711 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251027 14:55:06.678525 16711 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:55:06.678612 16711 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:06.678678 16711 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:06.678763 16711 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:55:06.678848 16711 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:55:06.689042 16711 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:06.690572 16711 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:06.691862 16711 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:06.699797 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:06.699903 16719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:06.699877 16717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:06.700911 16711 server_base.cc:1047] running on GCE node
I20251027 14:55:06.701586 16711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:06.702898 16711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:06.704092 16711 hybrid_clock.cc:648] HybridClock initialized: now 1761576906704045 us; error 39 us; skew 500 ppm
I20251027 14:55:06.704430 16711 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:06 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576906, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:06.707279 16711 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:55:06.708822 16711 webserver.cc:492] Webserver started at http://127.0.104.62:45481/ using document root <none> and password file <none>
I20251027 14:55:06.709489 16711 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:06.709590 16711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:06.709921 16711 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:55:06.712488 16711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "9739ec864cad41c2a9f68621e45e22b6"
format_stamp: "Formatted at 2025-10-27 14:55:06 on dist-test-slave-hxxt"
server_key: "0970753bc6e509e10558557031182cc0"
server_key_iv: "5775589f458c332f44348c6abc249df8"
server_key_version: "encryptionkey@0"
I20251027 14:55:06.713225 16711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "9739ec864cad41c2a9f68621e45e22b6"
format_stamp: "Formatted at 2025-10-27 14:55:06 on dist-test-slave-hxxt"
server_key: "0970753bc6e509e10558557031182cc0"
server_key_iv: "5775589f458c332f44348c6abc249df8"
server_key_version: "encryptionkey@0"
I20251027 14:55:06.718370 16711 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:55:06.721644 16726 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:06.722930 16711 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:55:06.723089 16711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "9739ec864cad41c2a9f68621e45e22b6"
format_stamp: "Formatted at 2025-10-27 14:55:06 on dist-test-slave-hxxt"
server_key: "0970753bc6e509e10558557031182cc0"
server_key_iv: "5775589f458c332f44348c6abc249df8"
server_key_version: "encryptionkey@0"
I20251027 14:55:06.723263 16711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:06.750983 16711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:06.755025 16711 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:06.755312 16711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:06.774456 16711 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37463
I20251027 14:55:06.774510 16777 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37463 every 8 connection(s)
I20251027 14:55:06.775921 16711 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:55:06.777318 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16711
I20251027 14:55:06.777521 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:55:06.778090 416 external_mini_cluster.cc:1468] Setting key 235a5f11eccf23cb2f727f5a1b3206ea
I20251027 14:55:06.780655 16778 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:06.793896 16778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap starting.
Oct 27 14:55:06 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:06.797633 16778 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:06.798744 16778 log.cc:826] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:06.802121 16778 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: No bootstrap required, opened a new log
I20251027 14:55:06.803887 16781 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:06.780153 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46122 (local address 127.0.104.62:37463)
1027 14:55:06.784968 (+ 4815us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:06.784980 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:06.785029 (+ 49us) server_negotiation.cc:408] Connection header received
1027 14:55:06.786181 (+ 1152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:06.786203 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:06.786638 (+ 435us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:06.787011 (+ 373us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:06.789001 (+ 1990us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:06.790409 (+ 1408us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:06.792075 (+ 1666us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:06.792636 (+ 561us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:06.796644 (+ 4008us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:06.796728 (+ 84us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:06.796744 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:06.796780 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:06.800045 (+ 3265us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:06.800904 (+ 859us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:06.800912 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:06.800919 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:06.801013 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:06.801337 (+ 324us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:06.801341 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:06.801343 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:06.801791 (+ 448us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:06.802041 (+ 250us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:06.802508 (+ 467us) server_negotiation.cc:300] Negotiation successful
1027 14:55:06.802917 (+ 409us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4601,"thread_start_us":4310,"threads_started":1}
I20251027 14:55:06.807499 16778 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:06.807803 16778 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:06.807904 16778 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9739ec864cad41c2a9f68621e45e22b6, State: Initialized, Role: FOLLOWER
I20251027 14:55:06.808573 16778 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:06.808794 16778 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:55:06.808914 16778 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:55:06.809063 16778 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:06.811282 16778 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:06.811782 16778 leader_election.cc:304] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9739ec864cad41c2a9f68621e45e22b6; no voters:
I20251027 14:55:06.812327 16778 leader_election.cc:290] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:55:06.812497 16783 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:55:06.813064 16783 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 LEADER]: Becoming Leader. State: Replica: 9739ec864cad41c2a9f68621e45e22b6, State: Running, Role: LEADER
I20251027 14:55:06.813553 16783 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:06.814205 16778 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:55:06.816489 16785 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9739ec864cad41c2a9f68621e45e22b6. Latest consensus state: current_term: 1 leader_uuid: "9739ec864cad41c2a9f68621e45e22b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } } }
I20251027 14:55:06.816798 16785 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: This master's current role is: LEADER
I20251027 14:55:06.817734 16790 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:55:06.817823 16784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9739ec864cad41c2a9f68621e45e22b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } } }
I20251027 14:55:06.818104 16784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: This master's current role is: LEADER
I20251027 14:55:06.823583 16790 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:55:06.832544 16790 catalog_manager.cc:1357] Generated new cluster ID: dd4315ed6b8d4f1f8488384b540ab6b6
I20251027 14:55:06.832666 16790 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:55:06.849964 16790 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:55:06.851107 16790 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:55:06.862330 16790 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Generated new TSK 0
I20251027 14:55:06.863122 16790 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:55:06.965107 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37463
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:07.292961 16806 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:07.293282 16806 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:07.293417 16806 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:07.301666 16806 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:07.301776 16806 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:07.301966 16806 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:55:07.312426 16806 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:07.313994 16806 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:07.315335 16806 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:07.324918 16811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:07.324942 16812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:07.325244 16814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:07.325971 16806 server_base.cc:1047] running on GCE node
I20251027 14:55:07.326524 16806 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:07.327384 16806 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:07.328595 16806 hybrid_clock.cc:648] HybridClock initialized: now 1761576907328518 us; error 45 us; skew 500 ppm
I20251027 14:55:07.328886 16806 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:07 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:07.331815 16806 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:55:07.333379 16806 webserver.cc:492] Webserver started at http://127.0.104.1:41283/ using document root <none> and password file <none>
I20251027 14:55:07.334125 16806 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:07.334218 16806 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:07.334558 16806 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:55:07.337148 16806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "6cdd8ffe904445fd96eee9be6fb96300"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "8ebf221173297bc71cc58e6ec705eb33"
server_key_iv: "342966c7e492a522770e809d81c04d6f"
server_key_version: "encryptionkey@0"
I20251027 14:55:07.337893 16806 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "6cdd8ffe904445fd96eee9be6fb96300"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "8ebf221173297bc71cc58e6ec705eb33"
server_key_iv: "342966c7e492a522770e809d81c04d6f"
server_key_version: "encryptionkey@0"
I20251027 14:55:07.343216 16806 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:55:07.346766 16821 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:07.348040 16806 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:55:07.348196 16806 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "6cdd8ffe904445fd96eee9be6fb96300"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "8ebf221173297bc71cc58e6ec705eb33"
server_key_iv: "342966c7e492a522770e809d81c04d6f"
server_key_version: "encryptionkey@0"
I20251027 14:55:07.348443 16806 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:07.363754 16806 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:07.367429 16806 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:07.367686 16806 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:07.368664 16806 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:07.370216 16806 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:07.370311 16806 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:07.370409 16806 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:07.370476 16806 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:07.400555 16806 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45585
I20251027 14:55:07.400574 16933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45585 every 8 connection(s)
I20251027 14:55:07.401913 16806 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:55:07.410102 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16806
I20251027 14:55:07.410298 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:55:07.410845 416 external_mini_cluster.cc:1468] Setting key a495083b590351ed36efa444ed2fc119
Oct 27 14:55:07 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:07.420485 16936 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:07.404542 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:44097 (local address 127.0.104.62:37463)
1027 14:55:07.404971 (+ 429us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:07.404977 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:07.405961 (+ 984us) server_negotiation.cc:408] Connection header received
1027 14:55:07.407475 (+ 1514us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:07.407482 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:07.407584 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:07.407686 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:07.409464 (+ 1778us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.410426 (+ 962us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:07.411551 (+ 1125us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.412019 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:07.414917 (+ 2898us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:07.414955 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:07.414960 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:07.414995 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:07.417490 (+ 2495us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:07.418216 (+ 726us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:07.418221 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:07.418223 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:07.418307 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:07.418647 (+ 340us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:07.418651 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:07.418653 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:07.418918 (+ 265us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:07.419073 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:07.419817 (+ 744us) server_negotiation.cc:300] Negotiation successful
1027 14:55:07.420085 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":322,"thread_start_us":202,"threads_started":1}
I20251027 14:55:07.421492 16937 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:07.405110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:44097)
1027 14:55:07.405805 (+ 695us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:07.405843 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:07.407190 (+ 1347us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:07.407966 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:07.407976 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:07.408586 (+ 610us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:07.409271 (+ 685us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:07.409290 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.410575 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:07.410579 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:07.411401 (+ 822us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:07.411413 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.411669 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:07.412465 (+ 796us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:07.412491 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:07.414697 (+ 2206us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:07.417643 (+ 2946us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:07.417653 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:07.417673 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:07.418087 (+ 414us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:07.418403 (+ 316us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:07.418406 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:07.418409 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:07.418536 (+ 127us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:07.419042 (+ 506us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:07.419051 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:07.419395 (+ 344us) client_negotiation.cc:769] Sending connection context
1027 14:55:07.419694 (+ 299us) client_negotiation.cc:240] Negotiation successful
1027 14:55:07.420223 (+ 529us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":294,"threads_started":1}
I20251027 14:55:07.423014 16934 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37463
I20251027 14:55:07.423398 16934 heartbeater.cc:461] Registering TS with master...
I20251027 14:55:07.424350 16934 heartbeater.cc:507] Master 127.0.104.62:37463 requested a full tablet report, sending...
I20251027 14:55:07.426792 16743 ts_manager.cc:194] Registered new tserver with Master: 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585)
I20251027 14:55:07.428467 16743 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:44097
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:55:07.501253 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37463
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:07.834389 16942 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:07.834733 16942 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:07.834873 16942 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:07.842986 16942 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:07.843103 16942 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:07.843279 16942 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:55:07.854189 16942 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:07.855777 16942 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:07.857108 16942 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:07.865845 16947 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:07.865845 16948 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:07.866465 16950 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:07.866775 16942 server_base.cc:1047] running on GCE node
I20251027 14:55:07.867304 16942 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:07.868131 16942 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:07.869334 16942 hybrid_clock.cc:648] HybridClock initialized: now 1761576907869241 us; error 68 us; skew 500 ppm
I20251027 14:55:07.869596 16942 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:07 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:07.872666 16942 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:55:07.874171 16942 webserver.cc:492] Webserver started at http://127.0.104.2:35089/ using document root <none> and password file <none>
I20251027 14:55:07.874876 16942 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:07.874965 16942 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:07.875324 16942 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:55:07.877890 16942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "316eeb2c599f436bad9b6870f494993e"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "c3d244e9494ed6c2fd4854efe5012056"
server_key_iv: "074aebb9b52498e220e5ce11667f8ab3"
server_key_version: "encryptionkey@0"
I20251027 14:55:07.878655 16942 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "316eeb2c599f436bad9b6870f494993e"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "c3d244e9494ed6c2fd4854efe5012056"
server_key_iv: "074aebb9b52498e220e5ce11667f8ab3"
server_key_version: "encryptionkey@0"
I20251027 14:55:07.883903 16942 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:55:07.887383 16957 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:07.888767 16942 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:55:07.888923 16942 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "316eeb2c599f436bad9b6870f494993e"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "c3d244e9494ed6c2fd4854efe5012056"
server_key_iv: "074aebb9b52498e220e5ce11667f8ab3"
server_key_version: "encryptionkey@0"
I20251027 14:55:07.889120 16942 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:07.910215 16942 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:07.913841 16942 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:07.914083 16942 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:07.915055 16942 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:07.916666 16942 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:07.916769 16942 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:07.916867 16942 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:07.916936 16942 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:07.948619 16942 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43391
I20251027 14:55:07.948673 17069 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43391 every 8 connection(s)
I20251027 14:55:07.949977 16942 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:55:07.954408 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 16942
I20251027 14:55:07.954617 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:55:07.955089 416 external_mini_cluster.cc:1468] Setting key e9f86ec36364fce8d7627ec5cf2b0a7c
Oct 27 14:55:07 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576907, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:07.969732 17072 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:07.952611 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:56181 (local address 127.0.104.62:37463)
1027 14:55:07.953029 (+ 418us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:07.953034 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:07.954083 (+ 1049us) server_negotiation.cc:408] Connection header received
1027 14:55:07.956035 (+ 1952us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:07.956042 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:07.956135 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:07.956238 (+ 103us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:07.958196 (+ 1958us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.959088 (+ 892us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:07.960473 (+ 1385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.960849 (+ 376us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:07.963843 (+ 2994us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:07.963879 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:07.963882 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:07.963911 (+ 29us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:07.966433 (+ 2522us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:07.967297 (+ 864us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:07.967301 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:07.967304 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:07.967377 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:07.967708 (+ 331us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:07.967712 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:07.967714 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:07.967995 (+ 281us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:07.968268 (+ 273us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:07.969118 (+ 850us) server_negotiation.cc:300] Negotiation successful
1027 14:55:07.969384 (+ 266us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":290,"thread_start_us":195,"threads_started":1}
I20251027 14:55:07.970661 17073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:07.953217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:56181)
1027 14:55:07.953919 (+ 702us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:07.953959 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:07.955733 (+ 1774us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:07.956510 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:07.956519 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:07.957084 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:07.958017 (+ 933us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:07.958042 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.959230 (+ 1188us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:07.959235 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:07.960306 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:07.960319 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:07.960595 (+ 276us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:07.961298 (+ 703us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:07.961328 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:07.963657 (+ 2329us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:07.966561 (+ 2904us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:07.966571 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:07.966589 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:07.967155 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:07.967476 (+ 321us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:07.967480 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:07.967482 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:07.967598 (+ 116us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:07.968235 (+ 637us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:07.968244 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:07.968633 (+ 389us) client_negotiation.cc:769] Sending connection context
1027 14:55:07.968982 (+ 349us) client_negotiation.cc:240] Negotiation successful
1027 14:55:07.969480 (+ 498us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":477,"thread_start_us":310,"threads_started":1}
I20251027 14:55:07.972121 17070 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37463
I20251027 14:55:07.972507 17070 heartbeater.cc:461] Registering TS with master...
I20251027 14:55:07.973315 17070 heartbeater.cc:507] Master 127.0.104.62:37463 requested a full tablet report, sending...
I20251027 14:55:07.974942 16743 ts_manager.cc:194] Registered new tserver with Master: 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391)
I20251027 14:55:07.976064 16743 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:56181
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:55:08.045312 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:37463
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:08.374114 17078 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:08.374459 17078 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:08.374595 17078 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:08.382712 17078 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:08.382820 17078 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:08.383004 17078 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:55:08.393550 17078 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:08.395157 17078 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:08.396500 17078 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:08.406471 17083 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:08.408062 17084 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:08.408667 17086 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:08.409359 17078 server_base.cc:1047] running on GCE node
I20251027 14:55:08.409937 17078 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:08.410862 17078 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:08.412075 17078 hybrid_clock.cc:648] HybridClock initialized: now 1761576908412008 us; error 41 us; skew 500 ppm
I20251027 14:55:08.412393 17078 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:08 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:08.415541 17078 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:55:08.417126 17078 webserver.cc:492] Webserver started at http://127.0.104.3:45159/ using document root <none> and password file <none>
I20251027 14:55:08.417832 17078 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:08.417917 17078 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:08.418289 17078 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:55:08.420989 17078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "76f2843631284d3a9d5dd82ca0107b6d"
format_stamp: "Formatted at 2025-10-27 14:55:08 on dist-test-slave-hxxt"
server_key: "dbb94d1af6f3ebde789c35d378e7fcba"
server_key_iv: "3af22ea873c95260588ef53417dfadbf"
server_key_version: "encryptionkey@0"
I20251027 14:55:08.421726 17078 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "76f2843631284d3a9d5dd82ca0107b6d"
format_stamp: "Formatted at 2025-10-27 14:55:08 on dist-test-slave-hxxt"
server_key: "dbb94d1af6f3ebde789c35d378e7fcba"
server_key_iv: "3af22ea873c95260588ef53417dfadbf"
server_key_version: "encryptionkey@0"
I20251027 14:55:08.426967 17078 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:55:08.430394 17093 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:08.431669 17078 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:55:08.431816 17078 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "76f2843631284d3a9d5dd82ca0107b6d"
format_stamp: "Formatted at 2025-10-27 14:55:08 on dist-test-slave-hxxt"
server_key: "dbb94d1af6f3ebde789c35d378e7fcba"
server_key_iv: "3af22ea873c95260588ef53417dfadbf"
server_key_version: "encryptionkey@0"
I20251027 14:55:08.432034 17078 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:08.432142 16934 heartbeater.cc:499] Master 127.0.104.62:37463 was elected leader, sending a full tablet report...
I20251027 14:55:08.449826 17078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:08.453711 17078 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:08.453966 17078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:08.454931 17078 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:08.456480 17078 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:08.456578 17078 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:08.456686 17078 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:08.456753 17078 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:08.486577 17078 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36255
I20251027 14:55:08.486615 17205 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36255 every 8 connection(s)
I20251027 14:55:08.487891 17078 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:55:08.496397 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17078
I20251027 14:55:08.496560 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:55:08.496985 416 external_mini_cluster.cc:1468] Setting key f1936730dcd9c1f452b61ff952cdd690
Oct 27 14:55:08 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576908, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:08.506181 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.490538 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:44705 (local address 127.0.104.62:37463)
1027 14:55:08.490944 (+ 406us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.490949 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.491945 (+ 996us) server_negotiation.cc:408] Connection header received
1027 14:55:08.493602 (+ 1657us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.493609 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.493701 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.493806 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:08.495483 (+ 1677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.496464 (+ 981us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.497651 (+ 1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.498131 (+ 480us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.500809 (+ 2678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:08.500844 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:08.500847 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:08.500878 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:08.503351 (+ 2473us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:08.504012 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:08.504017 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:08.504019 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:08.504088 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:08.504427 (+ 339us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:08.504433 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:08.504436 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:08.504671 (+ 235us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:08.504801 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.505540 (+ 739us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.505812 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":299,"thread_start_us":196,"threads_started":1}
I20251027 14:55:08.507025 17209 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.491064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:44705)
1027 14:55:08.491769 (+ 705us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:08.491809 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:08.493239 (+ 1430us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:08.494065 (+ 826us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:08.494074 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:08.494633 (+ 559us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:08.495302 (+ 669us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.495330 (+ 28us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.496614 (+ 1284us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.496618 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:08.497514 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.497526 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.497783 (+ 257us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.498483 (+ 700us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:08.498508 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:08.500623 (+ 2115us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:08.503486 (+ 2863us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:08.503495 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:08.503515 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:08.503895 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:08.504193 (+ 298us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:08.504196 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:08.504199 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:08.504316 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:08.504785 (+ 469us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:08.504792 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:08.505139 (+ 347us) client_negotiation.cc:769] Sending connection context
1027 14:55:08.505446 (+ 307us) client_negotiation.cc:240] Negotiation successful
1027 14:55:08.505928 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":479,"thread_start_us":314,"threads_started":1}
I20251027 14:55:08.508272 17206 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37463
I20251027 14:55:08.508641 17206 heartbeater.cc:461] Registering TS with master...
I20251027 14:55:08.509418 17206 heartbeater.cc:507] Master 127.0.104.62:37463 requested a full tablet report, sending...
I20251027 14:55:08.510946 16743 ts_manager.cc:194] Registered new tserver with Master: 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255)
I20251027 14:55:08.511917 16743 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:44705
I20251027 14:55:08.512603 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:55:08.528535 17208 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.516938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46134 (local address 127.0.104.62:37463)
1027 14:55:08.517102 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.517108 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.517452 (+ 344us) server_negotiation.cc:408] Connection header received
1027 14:55:08.517608 (+ 156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.517613 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.517672 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.517749 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:08.519305 (+ 1556us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.520162 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.521635 (+ 1473us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.522089 (+ 454us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.523625 (+ 1536us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:08.523662 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:08.523666 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:08.523693 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:08.526009 (+ 2316us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:08.526670 (+ 661us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:08.526675 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:08.526677 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:08.526749 (+ 72us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:08.527078 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:08.527083 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:08.527087 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:08.527362 (+ 275us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:08.527588 (+ 226us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.527939 (+ 351us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.528197 (+ 258us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251027 14:55:08.531948 16743 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46134:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:55:08.536548 16743 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:55:08.558203 17219 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.548593 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36255 (local address 127.0.0.1:54304)
1027 14:55:08.549237 (+ 644us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:08.549274 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:08.549464 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:08.552637 (+ 3173us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:08.552642 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:08.552667 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:08.553514 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.553527 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.555252 (+ 1725us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.555256 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:08.557232 (+ 1976us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.557247 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.557428 (+ 181us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.557503 (+ 75us) client_negotiation.cc:769] Sending connection context
1027 14:55:08.557648 (+ 145us) client_negotiation.cc:240] Negotiation successful
1027 14:55:08.557826 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":405,"thread_start_us":715,"threads_started":1}
I20251027 14:55:08.558887 17223 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.550471 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:43391 (local address 127.0.0.1:56780)
1027 14:55:08.551403 (+ 932us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:08.551419 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:08.551515 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:08.552941 (+ 1426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:08.552945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:08.552974 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:08.554059 (+ 1085us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.554072 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.555963 (+ 1891us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.555967 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:08.557532 (+ 1565us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.557544 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.557702 (+ 158us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.557727 (+ 25us) client_negotiation.cc:769] Sending connection context
1027 14:55:08.557793 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:55:08.557900 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":841,"thread_start_us":683,"threads_started":1}
I20251027 14:55:08.559417 17220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.548872 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:45972)
1027 14:55:08.550398 (+ 1526us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:08.550414 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:08.550536 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:08.551905 (+ 1369us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:08.551912 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:08.551949 (+ 37us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:08.552699 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.552718 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.555399 (+ 2681us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.555405 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:08.557231 (+ 1826us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.557247 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.558904 (+ 1657us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.558936 (+ 32us) client_negotiation.cc:769] Sending connection context
1027 14:55:08.559008 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:55:08.559128 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1412,"thread_start_us":1255,"threads_started":1}
I20251027 14:55:08.559599 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.550553 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56780 (local address 127.0.104.2:43391)
1027 14:55:08.552295 (+ 1742us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.552302 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.552324 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:55:08.552441 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.552449 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.552637 (+ 188us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.552843 (+ 206us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:08.554256 (+ 1413us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.555758 (+ 1502us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.557856 (+ 2098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.558880 (+ 1024us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.559032 (+ 152us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.559181 (+ 149us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.559356 (+ 175us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1617,"thread_start_us":1503,"threads_started":1}
I20251027 14:55:08.559782 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.549042 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54304 (local address 127.0.104.3:36255)
1027 14:55:08.552004 (+ 2962us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.552012 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.552044 (+ 32us) server_negotiation.cc:408] Connection header received
1027 14:55:08.552148 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.552157 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.552438 (+ 281us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.552643 (+ 205us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:08.553724 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.555063 (+ 1339us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.558003 (+ 2940us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.559094 (+ 1091us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.559233 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.559358 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.559538 (+ 180us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2815,"thread_start_us":2664,"threads_started":1}
I20251027 14:55:08.559974 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.549176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45972 (local address 127.0.104.1:45585)
1027 14:55:08.550058 (+ 882us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.550068 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.550679 (+ 611us) server_negotiation.cc:408] Connection header received
1027 14:55:08.550807 (+ 128us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.550817 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.551183 (+ 366us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.551500 (+ 317us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:08.552920 (+ 1420us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.555191 (+ 2271us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.557435 (+ 2244us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.558474 (+ 1039us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.559439 (+ 965us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.559562 (+ 123us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.559732 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":649,"threads_started":1}
I20251027 14:55:08.563536 16869 tablet_service.cc:1505] Processing CreateTablet for tablet f204de50e7a942d6813f0f2319950879 (DEFAULT_TABLE table=test-table [id=e385725a367f45aaa40f2fc19873534a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:55:08.563863 17005 tablet_service.cc:1505] Processing CreateTablet for tablet f204de50e7a942d6813f0f2319950879 (DEFAULT_TABLE table=test-table [id=e385725a367f45aaa40f2fc19873534a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:55:08.563863 17141 tablet_service.cc:1505] Processing CreateTablet for tablet f204de50e7a942d6813f0f2319950879 (DEFAULT_TABLE table=test-table [id=e385725a367f45aaa40f2fc19873534a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:55:08.564924 16869 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f204de50e7a942d6813f0f2319950879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:08.565305 17141 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f204de50e7a942d6813f0f2319950879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:08.565296 17005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f204de50e7a942d6813f0f2319950879. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:08.578027 17225 tablet_bootstrap.cc:492] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Bootstrap starting.
I20251027 14:55:08.580220 17226 tablet_bootstrap.cc:492] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Bootstrap starting.
I20251027 14:55:08.581152 17227 tablet_bootstrap.cc:492] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Bootstrap starting.
I20251027 14:55:08.581944 17225 tablet_bootstrap.cc:654] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:08.583398 17225 log.cc:826] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:08.584251 17226 tablet_bootstrap.cc:654] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:08.584688 17227 tablet_bootstrap.cc:654] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:08.585414 17226 log.cc:826] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:08.586124 17227 log.cc:826] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:08.588366 17226 tablet_bootstrap.cc:492] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: No bootstrap required, opened a new log
I20251027 14:55:08.588718 17226 ts_tablet_manager.cc:1403] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20251027 14:55:08.589171 17227 tablet_bootstrap.cc:492] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: No bootstrap required, opened a new log
I20251027 14:55:08.589452 17227 ts_tablet_manager.cc:1403] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.005s
I20251027 14:55:08.590190 17225 tablet_bootstrap.cc:492] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: No bootstrap required, opened a new log
I20251027 14:55:08.590472 17225 ts_tablet_manager.cc:1403] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Time spent bootstrapping tablet: real 0.013s user 0.008s sys 0.000s
I20251027 14:55:08.595019 17226 raft_consensus.cc:359] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.595353 17226 raft_consensus.cc:385] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:08.595422 17226 raft_consensus.cc:740] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316eeb2c599f436bad9b6870f494993e, State: Initialized, Role: FOLLOWER
I20251027 14:55:08.596055 17226 consensus_queue.cc:260] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.597167 17070 heartbeater.cc:499] Master 127.0.104.62:37463 was elected leader, sending a full tablet report...
I20251027 14:55:08.597647 17226 ts_tablet_manager.cc:1434] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20251027 14:55:08.597488 17225 raft_consensus.cc:359] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.596747 17227 raft_consensus.cc:359] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.597880 17225 raft_consensus.cc:385] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:08.598001 17227 raft_consensus.cc:385] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:08.598050 17225 raft_consensus.cc:740] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd8ffe904445fd96eee9be6fb96300, State: Initialized, Role: FOLLOWER
I20251027 14:55:08.598127 17227 raft_consensus.cc:740] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f2843631284d3a9d5dd82ca0107b6d, State: Initialized, Role: FOLLOWER
I20251027 14:55:08.598672 17225 consensus_queue.cc:260] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.598759 17227 consensus_queue.cc:260] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.600278 17225 ts_tablet_manager.cc:1434] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Time spent starting tablet: real 0.010s user 0.011s sys 0.000s
I20251027 14:55:08.601297 17206 heartbeater.cc:499] Master 127.0.104.62:37463 was elected leader, sending a full tablet report...
I20251027 14:55:08.601912 17227 ts_tablet_manager.cc:1434] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Time spent starting tablet: real 0.012s user 0.009s sys 0.002s
W20251027 14:55:08.654130 16935 tablet.cc:2378] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:55:08.701968 17071 tablet.cc:2378] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:55:08.739475 17207 tablet.cc:2378] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:55:08.871189 17231 raft_consensus.cc:493] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:55:08.871476 17231 raft_consensus.cc:515] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.873239 17231 leader_election.cc:290] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255), 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585)
I20251027 14:55:08.878696 17235 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.873561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36255 (local address 127.0.104.2:53239)
1027 14:55:08.874367 (+ 806us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:08.874383 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:08.874506 (+ 123us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:08.874796 (+ 290us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:08.874801 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:08.874829 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:08.875505 (+ 676us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.875515 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.876771 (+ 1256us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.876775 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:08.877968 (+ 1193us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.877980 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.878125 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.878157 (+ 32us) client_negotiation.cc:769] Sending connection context
1027 14:55:08.878224 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:55:08.878354 (+ 130us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":718,"thread_start_us":579,"threads_started":1}
I20251027 14:55:08.878726 17234 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.873541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.104.2:48327)
1027 14:55:08.874216 (+ 675us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:08.874232 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:08.874400 (+ 168us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:08.874733 (+ 333us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:08.874739 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:08.874764 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:08.875508 (+ 744us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.875519 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.876804 (+ 1285us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.876808 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:08.877989 (+ 1181us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:08.878001 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.878158 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.878182 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:55:08.878245 (+ 63us) client_negotiation.cc:240] Negotiation successful
1027 14:55:08.878355 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":559,"spinlock_wait_cycles":2304,"thread_start_us":346,"threads_started":1}
I20251027 14:55:08.879534 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.873537 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:53239 (local address 127.0.104.3:36255)
1027 14:55:08.873718 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.873724 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.874479 (+ 755us) server_negotiation.cc:408] Connection header received
1027 14:55:08.874615 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.874619 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.874681 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.874770 (+ 89us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:08.875640 (+ 870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.876589 (+ 949us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.878120 (+ 1531us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.879163 (+ 1043us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.879216 (+ 53us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.879290 (+ 74us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.879379 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:55:08.879566 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.873847 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:48327 (local address 127.0.104.1:45585)
1027 14:55:08.874006 (+ 159us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.874012 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.874350 (+ 338us) server_negotiation.cc:408] Connection header received
1027 14:55:08.874552 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.874558 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.874633 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.874698 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:08.875653 (+ 955us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.876608 (+ 955us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.878144 (+ 1536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.879171 (+ 1027us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.879222 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.879301 (+ 79us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.879399 (+ 98us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:55:08.880366 17161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f204de50e7a942d6813f0f2319950879" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f2843631284d3a9d5dd82ca0107b6d" is_pre_election: true
I20251027 14:55:08.880368 16889 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f204de50e7a942d6813f0f2319950879" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdd8ffe904445fd96eee9be6fb96300" is_pre_election: true
I20251027 14:55:08.880841 16889 raft_consensus.cc:2468] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 0.
I20251027 14:55:08.880839 17161 raft_consensus.cc:2468] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 0.
I20251027 14:55:08.881443 16958 leader_election.cc:304] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d; no voters:
I20251027 14:55:08.881783 17231 raft_consensus.cc:2804] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:55:08.881903 17231 raft_consensus.cc:493] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:55:08.882001 17231 raft_consensus.cc:3060] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:08.884218 17231 raft_consensus.cc:515] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.884815 17231 leader_election.cc:290] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 election: Requested vote from peers 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255), 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585)
I20251027 14:55:08.885238 17161 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f204de50e7a942d6813f0f2319950879" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f2843631284d3a9d5dd82ca0107b6d"
I20251027 14:55:08.885283 16889 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f204de50e7a942d6813f0f2319950879" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdd8ffe904445fd96eee9be6fb96300"
I20251027 14:55:08.885409 17161 raft_consensus.cc:3060] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:08.885437 16889 raft_consensus.cc:3060] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:08.887665 16889 raft_consensus.cc:2468] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 1.
I20251027 14:55:08.887881 17161 raft_consensus.cc:2468] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 1.
I20251027 14:55:08.888075 16961 leader_election.cc:304] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316eeb2c599f436bad9b6870f494993e, 6cdd8ffe904445fd96eee9be6fb96300; no voters:
I20251027 14:55:08.888326 17231 raft_consensus.cc:2804] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:55:08.888770 17231 raft_consensus.cc:697] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 1 LEADER]: Becoming Leader. State: Replica: 316eeb2c599f436bad9b6870f494993e, State: Running, Role: LEADER
I20251027 14:55:08.889189 17231 consensus_queue.cc:237] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:08.893828 16741 catalog_manager.cc:5649] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e reported cstate change: term changed from 0 to 1, leader changed from <none> to 316eeb2c599f436bad9b6870f494993e (127.0.104.2). New cstate: current_term: 1 leader_uuid: "316eeb2c599f436bad9b6870f494993e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } health_report { overall_health: HEALTHY } } }
I20251027 14:55:08.930701 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.925718 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56794 (local address 127.0.104.2:43391)
1027 14:55:08.925863 (+ 145us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.925868 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:08.925905 (+ 37us) server_negotiation.cc:408] Connection header received
1027 14:55:08.926107 (+ 202us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:08.926111 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:08.926236 (+ 125us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:08.926379 (+ 143us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:55:08.926904 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.927903 (+ 999us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:08.929151 (+ 1248us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:08.929491 (+ 340us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:08.929641 (+ 150us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:55:08.930177 (+ 536us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:55:08.930320 (+ 143us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:08.930412 (+ 92us) server_negotiation.cc:300] Negotiation successful
1027 14:55:08.930496 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251027 14:55:08.938668 16889 raft_consensus.cc:1275] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Refusing update from remote peer 316eeb2c599f436bad9b6870f494993e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:55:08.938668 17161 raft_consensus.cc:1275] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Refusing update from remote peer 316eeb2c599f436bad9b6870f494993e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:55:08.939738 17236 consensus_queue.cc:1048] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:55:08.939998 17231 consensus_queue.cc:1048] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:55:08.960490 17239 mvcc.cc:204] Tried to move back new op lower bound from 7215419019001860096 to 7215419018812964864. Current Snapshot: MvccSnapshot[applied={T|T < 7215419019001860096 or (T in {7215419019001860096})}]
I20251027 14:55:08.967784 16742 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46134:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:55:08.968040 16742 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:46134:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:55:08.970216 17241 mvcc.cc:204] Tried to move back new op lower bound from 7215419019001860096 to 7215419018812964864. Current Snapshot: MvccSnapshot[applied={T|T < 7215419019001860096}]
I20251027 14:55:08.973140 16742 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Sending DeleteTablet for 3 replicas of tablet f204de50e7a942d6813f0f2319950879
I20251027 14:55:08.974934 17141 tablet_service.cc:1552] Processing DeleteTablet for tablet f204de50e7a942d6813f0f2319950879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:55:08 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:54304
I20251027 14:55:08.977722 16869 tablet_service.cc:1552] Processing DeleteTablet for tablet f204de50e7a942d6813f0f2319950879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:55:08 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:45972
I20251027 14:55:08.978282 17005 tablet_service.cc:1552] Processing DeleteTablet for tablet f204de50e7a942d6813f0f2319950879 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:55:08 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:56780
I20251027 14:55:08.986186 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 16806
I20251027 14:55:09.005354 17251 tablet_replica.cc:333] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: stopping tablet replica
I20251027 14:55:09.005899 17251 raft_consensus.cc:2243] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:55:09.006779 17251 raft_consensus.cc:2272] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:55:09.008085 17249 tablet_replica.cc:333] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: stopping tablet replica
I20251027 14:55:09.008562 17249 raft_consensus.cc:2243] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:55:09.009058 17249 raft_consensus.cc:2272] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:55:09.011171 17250 tablet_replica.cc:333] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: stopping tablet replica
I20251027 14:55:09.011633 17250 raft_consensus.cc:2243] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:55:09.012022 17250 raft_consensus.cc:2272] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:55:09.012571 17249 ts_tablet_manager.cc:1916] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:55:09.012575 17251 ts_tablet_manager.cc:1916] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:55:09.016538 17250 ts_tablet_manager.cc:1916] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Deleting tablet data with delete state TABLET_DATA_DELETED
Oct 27 14:55:09 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:55:09.019572 17249 ts_tablet_manager.cc:1929] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:55:09.019807 17249 log.cc:1199] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/f204de50e7a942d6813f0f2319950879
I20251027 14:55:09.020356 17249 ts_tablet_manager.cc:1950] T f204de50e7a942d6813f0f2319950879 P 76f2843631284d3a9d5dd82ca0107b6d: Deleting consensus metadata
I20251027 14:55:09.022408 16727 catalog_manager.cc:4985] TS 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255): tablet f204de50e7a942d6813f0f2319950879 (table test-table [id=e385725a367f45aaa40f2fc19873534a]) successfully deleted
I20251027 14:55:09.024602 17251 ts_tablet_manager.cc:1929] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:55:09.024863 17251 log.cc:1199] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/f204de50e7a942d6813f0f2319950879
I20251027 14:55:09.025445 17251 ts_tablet_manager.cc:1950] T f204de50e7a942d6813f0f2319950879 P 316eeb2c599f436bad9b6870f494993e: Deleting consensus metadata
I20251027 14:55:09.027346 16728 catalog_manager.cc:4985] TS 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391): tablet f204de50e7a942d6813f0f2319950879 (table test-table [id=e385725a367f45aaa40f2fc19873534a]) successfully deleted
I20251027 14:55:09.027813 17222 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:08.987572 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45974 (local address 127.0.104.1:45585)
1027 14:55:08.987751 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:08.987757 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:09.002872 (+ 15115us) server_negotiation.cc:408] Connection header received
1027 14:55:09.003455 (+ 583us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:09.003461 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:09.003555 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:09.003632 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:09.006340 (+ 2708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:09.007864 (+ 1524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:09.011005 (+ 3141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:09.011563 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:09.020125 (+ 8562us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:09.020178 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:09.020196 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:09.020247 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:09.024223 (+ 3976us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:09.025191 (+ 968us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:09.025201 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:09.025205 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:09.025319 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:09.025796 (+ 477us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:09.025805 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:09.025808 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:09.026247 (+ 439us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:09.026539 (+ 292us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:09.027185 (+ 646us) server_negotiation.cc:300] Negotiation successful
1027 14:55:09.027457 (+ 272us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251027 14:55:09.028430 17250 ts_tablet_manager.cc:1929] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:55:09.028589 17250 log.cc:1199] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/f204de50e7a942d6813f0f2319950879
I20251027 14:55:09.028951 16929 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:09.118958 17250 ts_tablet_manager.cc:1950] T f204de50e7a942d6813f0f2319950879 P 6cdd8ffe904445fd96eee9be6fb96300: Deleting consensus metadata
I20251027 14:55:09.119431 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 16806
I20251027 14:55:09.134857 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 16942
W20251027 14:55:09.135042 16730 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:45585 (error 108)
I20251027 14:55:09.137560 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.136815 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:45980)
1027 14:55:09.137222 (+ 407us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.137353 (+ 131us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":198,"threads_started":1}
W20251027 14:55:09.137840 16730 catalog_manager.cc:4712] TS 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet f204de50e7a942d6813f0f2319950879: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Oct 27 14:55:09 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:55:09.150447 17224 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.135617 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:56808 (local address 127.0.104.2:43391)
1027 14:55:09.135795 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:09.135802 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:09.135920 (+ 118us) server_negotiation.cc:408] Connection header received
1027 14:55:09.136092 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:09.136099 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:09.136186 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:09.136283 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:09.138296 (+ 2013us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:09.139222 (+ 926us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:09.140547 (+ 1325us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:09.141055 (+ 508us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:09.144460 (+ 3405us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:09.144505 (+ 45us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:09.144521 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:09.144560 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:09.147576 (+ 3016us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:09.148304 (+ 728us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:09.148311 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:09.148313 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:09.148456 (+ 143us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:09.148899 (+ 443us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:09.148903 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:09.148905 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:09.149223 (+ 318us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:09.149534 (+ 311us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:09.149873 (+ 339us) server_negotiation.cc:300] Negotiation successful
1027 14:55:09.150130 (+ 257us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":42}
I20251027 14:55:09.151131 17065 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:09.191370 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.190870 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:45990)
1027 14:55:09.191128 (+ 258us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.191221 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":100}
I20251027 14:55:09.192351 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.192046 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:45992)
1027 14:55:09.192155 (+ 109us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.192219 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
I20251027 14:55:09.252898 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.252342 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46008)
1027 14:55:09.252614 (+ 272us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.252736 (+ 122us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":131}
I20251027 14:55:09.253894 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.253537 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46010)
1027 14:55:09.253675 (+ 138us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.253751 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":43}
I20251027 14:55:09.264261 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 16942
I20251027 14:55:09.279847 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 17078
Oct 27 14:55:09 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576906, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:55:09.293370 17221 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.280607 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54314 (local address 127.0.104.3:36255)
1027 14:55:09.280757 (+ 150us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:09.280762 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:09.280779 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:55:09.280919 (+ 140us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:09.280924 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:09.280989 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:09.281067 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:09.282490 (+ 1423us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:09.283347 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:09.284546 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:09.284877 (+ 331us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:09.288502 (+ 3625us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:09.288540 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:09.288556 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:09.288586 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:09.291188 (+ 2602us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:09.291787 (+ 599us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:09.291795 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:09.291797 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:09.291877 (+ 80us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:09.292095 (+ 218us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:09.292098 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:09.292100 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:09.292372 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:09.292655 (+ 283us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:09.292874 (+ 219us) server_negotiation.cc:300] Negotiation successful
1027 14:55:09.293052 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:55:09.294001 17201 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:09.334254 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.333835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46018)
1027 14:55:09.334026 (+ 191us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.334112 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20251027 14:55:09.335366 17255 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:09.334970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46022)
1027 14:55:09.335150 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:09.335218 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":97}
I20251027 14:55:09.381964 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 17078
I20251027 14:55:09.395864 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 16711
I20251027 14:55:09.396987 16773 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:09.507339 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 16711
I20251027 14:55:29.525761 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37463
--webserver_interface=127.0.104.62
--webserver_port=45481
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--tsk_private_key_password_cmd=echo foo with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:29.859104 17259 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:29.859413 17259 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:29.859542 17259 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:29.867561 17259 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251027 14:55:29.867648 17259 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251027 14:55:29.867703 17259 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251027 14:55:29.867760 17259 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:55:29.867847 17259 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:29.867918 17259 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:29.868002 17259 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:55:29.868095 17259 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:55:29.878652 17259 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_private_key_password_cmd=echo foo
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=45481
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:29.880277 17259 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:29.881639 17259 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:29.900530 17268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:29.900605 17265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:29.900549 17266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:29.901579 17259 server_base.cc:1047] running on GCE node
I20251027 14:55:29.902495 17259 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:29.904135 17259 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:29.905431 17259 hybrid_clock.cc:648] HybridClock initialized: now 1761576929905363 us; error 40 us; skew 500 ppm
I20251027 14:55:29.905929 17259 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:29 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576929, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:29.909646 17259 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:55:29.911573 17259 webserver.cc:492] Webserver started at http://127.0.104.62:45481/ using document root <none> and password file <none>
I20251027 14:55:29.912516 17259 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:29.912657 17259 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:29.919160 17259 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251027 14:55:29.922333 17275 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:29.923761 17259 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:55:29.923944 17259 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "9739ec864cad41c2a9f68621e45e22b6"
format_stamp: "Formatted at 2025-10-27 14:55:06 on dist-test-slave-hxxt"
server_key: "0970753bc6e509e10558557031182cc0"
server_key_iv: "5775589f458c332f44348c6abc249df8"
server_key_version: "encryptionkey@0"
I20251027 14:55:29.924823 17259 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:29.956919 17259 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:29.960996 17259 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:29.961262 17259 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:29.980733 17259 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37463
I20251027 14:55:29.980746 17326 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37463 every 8 connection(s)
I20251027 14:55:29.982308 17259 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:55:29.988804 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17259
I20251027 14:55:29.989630 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:45585
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:37463
--webserver_port=41283
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:55:29.989965 17327 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:55:29.993412 17327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap starting.
I20251027 14:55:30.017599 17327 log.cc:826] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:30.032940 17327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=8 overwritten=0 applied=8 ignored=0} inserts{seen=5 ignored=0} mutations{seen=4 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:55:30.033470 17327 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap complete.
I20251027 14:55:30.040093 17327 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:30.040798 17327 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9739ec864cad41c2a9f68621e45e22b6, State: Initialized, Role: FOLLOWER
I20251027 14:55:30.041492 17327 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:30.041692 17327 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:55:30.041805 17327 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:55:30.041961 17327 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 1 FOLLOWER]: Advancing to term 2
I20251027 14:55:30.045595 17327 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:30.046077 17327 leader_election.cc:304] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9739ec864cad41c2a9f68621e45e22b6; no voters:
I20251027 14:55:30.046653 17327 leader_election.cc:290] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [CANDIDATE]: Term 2 election: Requested vote from peers
I20251027 14:55:30.046851 17331 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Leader election won for term 2
I20251027 14:55:30.047715 17331 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 LEADER]: Becoming Leader. State: Replica: 9739ec864cad41c2a9f68621e45e22b6, State: Running, Role: LEADER
I20251027 14:55:30.048556 17327 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:55:30.048449 17331 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 8, Committed index: 8, Last appended: 1.8, Last appended by leader: 8, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:30.051853 17333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9739ec864cad41c2a9f68621e45e22b6. Latest consensus state: current_term: 2 leader_uuid: "9739ec864cad41c2a9f68621e45e22b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } } }
I20251027 14:55:30.052229 17333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: This master's current role is: LEADER
I20251027 14:55:30.052780 17332 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "9739ec864cad41c2a9f68621e45e22b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } } }
I20251027 14:55:30.052996 17332 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: This master's current role is: LEADER
I20251027 14:55:30.060178 17340 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:55:30.068517 17340 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:55:30.069959 17340 catalog_manager.cc:1269] Loaded cluster ID: dd4315ed6b8d4f1f8488384b540ab6b6
I20251027 14:55:30.070158 17340 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:55:30.074313 17340 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:55:30.076015 17340 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Loaded TSK: 0
I20251027 14:55:30.101424 17340 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Generated new TSK 1
I20251027 14:55:30.101847 17340 catalog_manager.cc:1524] Initializing in-progress tserver states...
W20251027 14:55:30.346182 17329 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:30.346493 17329 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:30.346621 17329 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:30.355135 17329 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:30.355239 17329 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:30.355436 17329 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:55:30.365825 17329 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:45585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=41283
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:30.367401 17329 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:30.368835 17329 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:30.378103 17355 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:30.378619 17356 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:30.379159 17358 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:30.380335 17329 server_base.cc:1047] running on GCE node
I20251027 14:55:30.380980 17329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:30.381956 17329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:30.383190 17329 hybrid_clock.cc:648] HybridClock initialized: now 1761576930383120 us; error 51 us; skew 500 ppm
I20251027 14:55:30.383479 17329 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:30 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:30.386669 17329 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:55:30.388397 17329 webserver.cc:492] Webserver started at http://127.0.104.1:41283/ using document root <none> and password file <none>
I20251027 14:55:30.389163 17329 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:30.389263 17329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:30.394984 17329 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251027 14:55:30.397883 17365 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:30.399679 17329 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:55:30.399852 17329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "6cdd8ffe904445fd96eee9be6fb96300"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "8ebf221173297bc71cc58e6ec705eb33"
server_key_iv: "342966c7e492a522770e809d81c04d6f"
server_key_version: "encryptionkey@0"
I20251027 14:55:30.400823 17329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:30.424639 17329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:30.429569 17329 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:30.429860 17329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:30.431013 17329 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:30.432938 17329 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:30.433027 17329 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:30.433140 17329 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:30.433194 17329 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:30.464766 17329 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45585
I20251027 14:55:30.464799 17477 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45585 every 8 connection(s)
I20251027 14:55:30.466194 17329 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:55:30.467449 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17329
I20251027 14:55:30.468214 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:43391
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:37463
--webserver_port=35089
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:55:30 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:30.488796 17481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:30.469957 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:33263)
1027 14:55:30.470740 (+ 783us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:30.470776 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:30.472172 (+ 1396us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:30.473664 (+ 1492us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:30.473676 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:30.474335 (+ 659us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:30.475009 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:30.475028 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.476792 (+ 1764us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:30.476796 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:30.477621 (+ 825us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:30.477633 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.477889 (+ 256us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:30.478657 (+ 768us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:30.478682 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:30.480923 (+ 2241us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:30.485036 (+ 4113us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:30.485043 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:30.485056 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:30.485454 (+ 398us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:30.485876 (+ 422us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:30.485879 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:30.485881 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:30.485981 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:30.486815 (+ 834us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:30.486821 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:30.487105 (+ 284us) client_negotiation.cc:769] Sending connection context
1027 14:55:30.487393 (+ 288us) client_negotiation.cc:240] Negotiation successful
1027 14:55:30.487762 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":560,"thread_start_us":376,"threads_started":1}
I20251027 14:55:30.489414 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:30.469512 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:33263 (local address 127.0.104.62:37463)
1027 14:55:30.470038 (+ 526us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:30.470050 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:30.470922 (+ 872us) server_negotiation.cc:408] Connection header received
1027 14:55:30.472741 (+ 1819us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:30.472755 (+ 14us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:30.473110 (+ 355us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:30.473496 (+ 386us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:30.475214 (+ 1718us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.476633 (+ 1419us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:30.477760 (+ 1127us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.478290 (+ 530us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:30.481169 (+ 2879us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:30.481250 (+ 81us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:30.481273 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:30.481335 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:30.484884 (+ 3549us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:30.485593 (+ 709us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:30.485604 (+ 11us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:30.485613 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:30.485769 (+ 156us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:30.486069 (+ 300us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:30.486074 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:30.486078 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:30.486697 (+ 619us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:30.486906 (+ 209us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:30.487674 (+ 768us) server_negotiation.cc:300] Negotiation successful
1027 14:55:30.488147 (+ 473us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":356,"thread_start_us":254,"threads_started":1}
I20251027 14:55:30.492043 17478 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37463
I20251027 14:55:30.492476 17478 heartbeater.cc:461] Registering TS with master...
I20251027 14:55:30.493430 17478 heartbeater.cc:507] Master 127.0.104.62:37463 requested a full tablet report, sending...
I20251027 14:55:30.495767 17292 ts_manager.cc:194] Registered new tserver with Master: 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585)
I20251027 14:55:30.497478 17292 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:33263
W20251027 14:55:30.802196 17482 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:30.802497 17482 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:30.802644 17482 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:30.810778 17482 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:30.810885 17482 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:30.811074 17482 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:55:30.821622 17482 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=35089
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:30.823238 17482 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:30.824626 17482 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:30.833851 17488 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:30.834489 17487 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:30.834862 17490 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:30.835266 17482 server_base.cc:1047] running on GCE node
I20251027 14:55:30.835834 17482 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:30.836782 17482 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:30.838003 17482 hybrid_clock.cc:648] HybridClock initialized: now 1761576930837939 us; error 35 us; skew 500 ppm
I20251027 14:55:30.838310 17482 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:30 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:30.841425 17482 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:55:30.842914 17482 webserver.cc:492] Webserver started at http://127.0.104.2:35089/ using document root <none> and password file <none>
I20251027 14:55:30.843626 17482 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:30.843724 17482 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:30.849232 17482 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:55:30.851889 17497 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:30.853271 17482 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:55:30.853417 17482 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "316eeb2c599f436bad9b6870f494993e"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "c3d244e9494ed6c2fd4854efe5012056"
server_key_iv: "074aebb9b52498e220e5ce11667f8ab3"
server_key_version: "encryptionkey@0"
I20251027 14:55:30.854100 17482 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:30.877086 17482 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:30.880883 17482 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:30.881145 17482 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:30.882159 17482 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:30.883780 17482 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:30.883853 17482 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:30.883965 17482 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:30.884012 17482 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:30.913420 17482 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43391
I20251027 14:55:30.913492 17609 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43391 every 8 connection(s)
I20251027 14:55:30.914884 17482 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:55:30.923632 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17482
I20251027 14:55:30.924250 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:36255
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:37463
--webserver_port=45159
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
Oct 27 14:55:30 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576930, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:30.933876 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:30.917714 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:42075 (local address 127.0.104.62:37463)
1027 14:55:30.917896 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:30.917901 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:30.919173 (+ 1272us) server_negotiation.cc:408] Connection header received
1027 14:55:30.920778 (+ 1605us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:30.920785 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:30.920879 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:30.920966 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:30.922623 (+ 1657us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.923481 (+ 858us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:30.924731 (+ 1250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.925201 (+ 470us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:30.928125 (+ 2924us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:30.928165 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:30.928169 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:30.928200 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:30.930765 (+ 2565us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:30.931471 (+ 706us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:30.931476 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:30.931478 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:30.931562 (+ 84us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:30.931883 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:30.931889 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:30.931892 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:30.932135 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:30.932273 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:30.933284 (+ 1011us) server_negotiation.cc:300] Negotiation successful
1027 14:55:30.933572 (+ 288us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251027 14:55:30.935513 17612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:30.918241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:42075)
1027 14:55:30.918995 (+ 754us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:30.919033 (+ 38us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:30.920482 (+ 1449us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:30.921214 (+ 732us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:30.921223 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:30.921794 (+ 571us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:30.922452 (+ 658us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:30.922470 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.923639 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:30.923643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:30.924593 (+ 950us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:30.924605 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:30.924852 (+ 247us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:30.925651 (+ 799us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:30.925678 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:30.927914 (+ 2236us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:30.930913 (+ 2999us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:30.930922 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:30.930941 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:30.931353 (+ 412us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:30.931662 (+ 309us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:30.931666 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:30.931669 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:30.931782 (+ 113us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:30.932257 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:30.932265 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:30.932637 (+ 372us) client_negotiation.cc:769] Sending connection context
1027 14:55:30.934091 (+ 1454us) client_negotiation.cc:240] Negotiation successful
1027 14:55:30.934489 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":364,"threads_started":1}
I20251027 14:55:30.937049 17610 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37463
I20251027 14:55:30.937388 17610 heartbeater.cc:461] Registering TS with master...
I20251027 14:55:30.938200 17610 heartbeater.cc:507] Master 127.0.104.62:37463 requested a full tablet report, sending...
I20251027 14:55:30.939697 17292 ts_manager.cc:194] Registered new tserver with Master: 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391)
I20251027 14:55:30.940577 17292 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:42075
W20251027 14:55:31.248078 17613 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:31.248368 17613 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:31.248525 17613 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:31.256667 17613 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:31.256774 17613 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:31.256955 17613 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:55:31.267395 17613 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=45159
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:31.268960 17613 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:31.270259 17613 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:31.279235 17619 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:31.279275 17618 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:31.280150 17621 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:31.280570 17613 server_base.cc:1047] running on GCE node
I20251027 14:55:31.281158 17613 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:31.282059 17613 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:31.283291 17613 hybrid_clock.cc:648] HybridClock initialized: now 1761576931283223 us; error 44 us; skew 500 ppm
I20251027 14:55:31.283604 17613 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:31 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:31.286851 17613 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:55:31.288337 17613 webserver.cc:492] Webserver started at http://127.0.104.3:45159/ using document root <none> and password file <none>
I20251027 14:55:31.289085 17613 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:31.289187 17613 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:31.294587 17613 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20251027 14:55:31.297367 17628 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:31.298799 17613 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20251027 14:55:31.298955 17613 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "76f2843631284d3a9d5dd82ca0107b6d"
format_stamp: "Formatted at 2025-10-27 14:55:08 on dist-test-slave-hxxt"
server_key: "dbb94d1af6f3ebde789c35d378e7fcba"
server_key_iv: "3af22ea873c95260588ef53417dfadbf"
server_key_version: "encryptionkey@0"
I20251027 14:55:31.299635 17613 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:31.336728 17613 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:31.340693 17613 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:31.340963 17613 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:31.341945 17613 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:31.343487 17613 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:31.343565 17613 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:31.343684 17613 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:31.343736 17613 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:31.374801 17613 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36255
I20251027 14:55:31.374843 17740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36255 every 8 connection(s)
I20251027 14:55:31.376263 17613 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:55:31.378228 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17613
Oct 27 14:55:31 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576931, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:55:31.393308 17480 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.379230 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42622 (local address 127.0.104.62:37463)
1027 14:55:31.379434 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.379441 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.380073 (+ 632us) server_negotiation.cc:408] Connection header received
1027 14:55:31.380243 (+ 170us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.380247 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.380315 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.380462 (+ 147us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:31.382493 (+ 2031us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.383881 (+ 1388us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.385393 (+ 1512us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.385752 (+ 359us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.387973 (+ 2221us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:31.388008 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:31.388014 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:31.388050 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:31.390721 (+ 2671us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.391369 (+ 648us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.391374 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.391376 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.391451 (+ 75us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.391758 (+ 307us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.391764 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.391766 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.392004 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:31.392316 (+ 312us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.392562 (+ 246us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.392907 (+ 345us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":88}
I20251027 14:55:31.396303 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.379283 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:53797 (local address 127.0.104.62:37463)
1027 14:55:31.379720 (+ 437us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.379725 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.380970 (+ 1245us) server_negotiation.cc:408] Connection header received
1027 14:55:31.382492 (+ 1522us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.382497 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.382569 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.382663 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:31.384667 (+ 2004us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.386035 (+ 1368us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.387123 (+ 1088us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.387572 (+ 449us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.390423 (+ 2851us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:31.390509 (+ 86us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:31.390513 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:31.390554 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:31.393404 (+ 2850us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.394205 (+ 801us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.394211 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.394215 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.394288 (+ 73us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.394610 (+ 322us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.394616 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.394620 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.394863 (+ 243us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:31.394991 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.395691 (+ 700us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.395945 (+ 254us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":362,"thread_start_us":286,"threads_started":1}
I20251027 14:55:31.397228 17745 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.380018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:53797)
1027 14:55:31.380823 (+ 805us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.380860 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:31.382201 (+ 1341us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:31.382938 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:31.382950 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:31.383699 (+ 749us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:31.384495 (+ 796us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.384515 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.386176 (+ 1661us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.386180 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:31.386988 (+ 808us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.386999 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.387202 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.387787 (+ 585us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:31.387812 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:31.390223 (+ 2411us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:31.393580 (+ 3357us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:31.393587 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:31.393602 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:31.394087 (+ 485us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:31.394380 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:31.394384 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:31.394386 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:31.394503 (+ 117us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:31.394978 (+ 475us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:31.394987 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:31.395299 (+ 312us) client_negotiation.cc:769] Sending connection context
1027 14:55:31.395593 (+ 294us) client_negotiation.cc:240] Negotiation successful
1027 14:55:31.396081 (+ 488us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":563,"thread_start_us":374,"threads_started":1}
I20251027 14:55:31.398535 17741 heartbeater.cc:344] Connected to a master server at 127.0.104.62:37463
I20251027 14:55:31.398873 17741 heartbeater.cc:461] Registering TS with master...
I20251027 14:55:31.399595 17741 heartbeater.cc:507] Master 127.0.104.62:37463 requested a full tablet report, sending...
I20251027 14:55:31.400974 17292 ts_manager.cc:194] Registered new tserver with Master: 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255)
I20251027 14:55:31.401767 17292 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:53797
I20251027 14:55:31.407907 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:55:31.414232 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.408845 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42626 (local address 127.0.104.62:37463)
1027 14:55:31.409003 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.409008 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.409427 (+ 419us) server_negotiation.cc:408] Connection header received
1027 14:55:31.409574 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.409578 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.409729 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.409869 (+ 140us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:55:31.410551 (+ 682us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.411444 (+ 893us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.412840 (+ 1396us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.413253 (+ 413us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.413416 (+ 163us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:55:31.413704 (+ 288us) server_negotiation.cc:395] Sending RPC error: FATAL_INVALID_AUTHENTICATION_TOKEN: Not authorized: token expired
1027 14:55:31.414023 (+ 319us) negotiation.cc:327] Negotiation complete: Not authorized: Server connection negotiation failed: server connection from 127.0.0.1:42626: token expired
Metrics: {"server-negotiator.queue_time_us":60}
W20251027 14:55:31.414340 17744 negotiation.cc:344] Unauthorized connection attempt: Server connection negotiation failed: server connection from 127.0.0.1:42626: token expired
W20251027 14:55:31.414987 17213 master_proxy_rpc.cc:203] Re-attempting CreateTable request to leader Master (127.0.104.62:37463)
I20251027 14:55:31.426612 17744 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.415474 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42638 (local address 127.0.104.62:37463)
1027 14:55:31.415638 (+ 164us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.415642 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.415657 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:55:31.415770 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.415774 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.415831 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.415905 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:31.417495 (+ 1590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.418359 (+ 864us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.419627 (+ 1268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.420024 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.421868 (+ 1844us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:31.421892 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:31.421896 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:31.421926 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:31.424334 (+ 2408us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.425004 (+ 670us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.425008 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.425010 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.425097 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.425377 (+ 280us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.425382 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.425384 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.425690 (+ 306us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:31.425915 (+ 225us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.426197 (+ 282us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.426383 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":75}
I20251027 14:55:31.429852 17292 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42638:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:55:31.434826 17292 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:55:31.467319 17751 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.446835 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36255 (local address 127.0.0.1:51880)
1027 14:55:31.459590 (+ 12755us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.459609 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:31.459731 (+ 122us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:31.461028 (+ 1297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:31.461036 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:31.461069 (+ 33us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:31.461836 (+ 767us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.461851 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.464441 (+ 2590us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.464448 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:31.466308 (+ 1860us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.466322 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.466514 (+ 192us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.466585 (+ 71us) client_negotiation.cc:769] Sending connection context
1027 14:55:31.466742 (+ 157us) client_negotiation.cc:240] Negotiation successful
1027 14:55:31.466938 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12654,"thread_start_us":10535,"threads_started":1}
I20251027 14:55:31.468089 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.446529 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46534)
1027 14:55:31.457875 (+ 11346us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.457926 (+ 51us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:31.458157 (+ 231us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:31.461852 (+ 3695us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:31.461861 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:31.461904 (+ 43us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:31.462706 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.462716 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.465720 (+ 3004us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.465726 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:31.467451 (+ 1725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.467467 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.467629 (+ 162us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.467652 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:55:31.467711 (+ 59us) client_negotiation.cc:240] Negotiation successful
1027 14:55:31.467815 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11041,"spinlock_wait_cycles":768,"thread_start_us":12825,"threads_started":1}
I20251027 14:55:31.469540 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.449250 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53638 (local address 127.0.104.2:43391)
1027 14:55:31.460167 (+ 10917us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.460175 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.460699 (+ 524us) server_negotiation.cc:408] Connection header received
1027 14:55:31.460911 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.460919 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.461259 (+ 340us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.461497 (+ 238us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:31.462889 (+ 1392us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.464629 (+ 1740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.467696 (+ 3067us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.468835 (+ 1139us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.468984 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.469114 (+ 130us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.469288 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10800,"thread_start_us":10706,"threads_started":1}
I20251027 14:55:31.469863 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.447167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51880 (local address 127.0.104.3:36255)
1027 14:55:31.460083 (+ 12916us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.460094 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.460133 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:55:31.460250 (+ 117us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.460260 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.460773 (+ 513us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.462062 (+ 1289us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:31.462372 (+ 310us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.464241 (+ 1869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.467894 (+ 3653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.469181 (+ 1287us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.469308 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.469438 (+ 130us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.469598 (+ 160us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12736,"thread_start_us":13223,"threads_started":1}
I20251027 14:55:31.469676 17755 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.448658 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:43391 (local address 127.0.0.1:53638)
1027 14:55:31.460584 (+ 11926us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.460597 (+ 13us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:31.460703 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:31.461774 (+ 1071us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:31.461778 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:31.461803 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:31.462694 (+ 891us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.462706 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.464843 (+ 2137us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.464849 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:31.466700 (+ 1851us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.466724 (+ 24us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.466888 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.466916 (+ 28us) client_negotiation.cc:769] Sending connection context
1027 14:55:31.466980 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:55:31.467105 (+ 125us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11833,"thread_start_us":11665,"threads_started":1}
I20251027 14:55:31.470698 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.446735 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46534 (local address 127.0.104.1:45585)
1027 14:55:31.461084 (+ 14349us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.461092 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.461118 (+ 26us) server_negotiation.cc:408] Connection header received
1027 14:55:31.461216 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.461224 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.461435 (+ 211us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.461638 (+ 203us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:31.462862 (+ 1224us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.465466 (+ 2604us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.468359 (+ 2893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.469434 (+ 1075us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.469538 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.470267 (+ 729us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.470438 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14217,"thread_start_us":14121,"threads_started":1}
I20251027 14:55:31.474135 17545 tablet_service.cc:1505] Processing CreateTablet for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 (DEFAULT_TABLE table=test-table [id=d97b042f8a2d4efdb5e990e57ff6dd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:55:31.474334 17676 tablet_service.cc:1505] Processing CreateTablet for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 (DEFAULT_TABLE table=test-table [id=d97b042f8a2d4efdb5e990e57ff6dd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:55:31.474749 17413 tablet_service.cc:1505] Processing CreateTablet for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 (DEFAULT_TABLE table=test-table [id=d97b042f8a2d4efdb5e990e57ff6dd55]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:55:31.475368 17545 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:31.475720 17676 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:31.476064 17413 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:55:31.486668 17757 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Bootstrap starting.
I20251027 14:55:31.490578 17759 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Bootstrap starting.
I20251027 14:55:31.490717 17757 tablet_bootstrap.cc:654] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:31.491362 17758 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Bootstrap starting.
I20251027 14:55:31.492121 17757 log.cc:826] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:31.494544 17759 tablet_bootstrap.cc:654] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:31.494562 17758 tablet_bootstrap.cc:654] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Neither blocks nor log segments found. Creating new log.
I20251027 14:55:31.495491 17757 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: No bootstrap required, opened a new log
I20251027 14:55:31.495759 17759 log.cc:826] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:31.495827 17757 ts_tablet_manager.cc:1403] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20251027 14:55:31.495784 17758 log.cc:826] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:31.499002 17758 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: No bootstrap required, opened a new log
I20251027 14:55:31.499107 17759 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: No bootstrap required, opened a new log
I20251027 14:55:31.499301 17758 ts_tablet_manager.cc:1403] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Time spent bootstrapping tablet: real 0.008s user 0.003s sys 0.003s
I20251027 14:55:31.499378 17759 ts_tablet_manager.cc:1403] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Time spent bootstrapping tablet: real 0.009s user 0.007s sys 0.000s
I20251027 14:55:31.501067 17478 heartbeater.cc:499] Master 127.0.104.62:37463 was elected leader, sending a full tablet report...
I20251027 14:55:31.501765 17757 raft_consensus.cc:359] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.502081 17757 raft_consensus.cc:385] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:31.502207 17757 raft_consensus.cc:740] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 316eeb2c599f436bad9b6870f494993e, State: Initialized, Role: FOLLOWER
I20251027 14:55:31.502856 17757 consensus_queue.cc:260] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.506407 17758 raft_consensus.cc:359] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.506820 17758 raft_consensus.cc:385] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:31.506927 17758 raft_consensus.cc:740] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd8ffe904445fd96eee9be6fb96300, State: Initialized, Role: FOLLOWER
I20251027 14:55:31.507617 17758 consensus_queue.cc:260] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.509333 17758 ts_tablet_manager.cc:1434] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Time spent starting tablet: real 0.010s user 0.010s sys 0.001s
I20251027 14:55:31.509775 17759 raft_consensus.cc:359] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.510026 17610 heartbeater.cc:499] Master 127.0.104.62:37463 was elected leader, sending a full tablet report...
I20251027 14:55:31.510222 17759 raft_consensus.cc:385] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:55:31.510370 17759 raft_consensus.cc:740] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 76f2843631284d3a9d5dd82ca0107b6d, State: Initialized, Role: FOLLOWER
I20251027 14:55:31.511111 17757 ts_tablet_manager.cc:1434] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Time spent starting tablet: real 0.015s user 0.011s sys 0.004s
I20251027 14:55:31.511103 17759 consensus_queue.cc:260] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.512185 17741 heartbeater.cc:499] Master 127.0.104.62:37463 was elected leader, sending a full tablet report...
I20251027 14:55:31.512600 17759 ts_tablet_manager.cc:1434] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Time spent starting tablet: real 0.013s user 0.005s sys 0.004s
I20251027 14:55:31.625769 17763 raft_consensus.cc:493] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:55:31.626060 17763 raft_consensus.cc:515] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.627856 17763 leader_election.cc:290] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585), 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255)
W20251027 14:55:31.628209 17742 tablet.cc:2378] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:55:31.633967 17766 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.628158 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36255 (local address 127.0.104.2:50187)
1027 14:55:31.629213 (+ 1055us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.629228 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:31.629414 (+ 186us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:31.629721 (+ 307us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:31.629726 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:31.629753 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:31.630533 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.630546 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.631835 (+ 1289us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.631840 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:31.633151 (+ 1311us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.633171 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.633377 (+ 206us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.633410 (+ 33us) client_negotiation.cc:769] Sending connection context
1027 14:55:31.633482 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:55:31.633629 (+ 147us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":965,"thread_start_us":444,"threads_started":1}
I20251027 14:55:31.634804 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.628416 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:50187 (local address 127.0.104.3:36255)
1027 14:55:31.628577 (+ 161us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.628584 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.629321 (+ 737us) server_negotiation.cc:408] Connection header received
1027 14:55:31.629544 (+ 223us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.629549 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.629611 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.629687 (+ 76us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:31.630689 (+ 1002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.631608 (+ 919us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.633321 (+ 1713us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.634430 (+ 1109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.634474 (+ 44us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.634538 (+ 64us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.634632 (+ 94us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":36}
I20251027 14:55:31.634948 17767 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.628091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.104.2:57763)
1027 14:55:31.628905 (+ 814us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.628977 (+ 72us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:31.629174 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:31.629884 (+ 710us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:31.629891 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:31.629936 (+ 45us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:55:31.630589 (+ 653us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.630600 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.631870 (+ 1270us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.631883 (+ 13us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:31.633256 (+ 1373us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:31.633274 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.634397 (+ 1123us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.634421 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:55:31.634517 (+ 96us) client_negotiation.cc:240] Negotiation successful
1027 14:55:31.634661 (+ 144us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":694,"thread_start_us":902,"threads_started":1}
I20251027 14:55:31.634960 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.629362 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:57763 (local address 127.0.104.1:45585)
1027 14:55:31.629514 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.629519 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.629536 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:55:31.629598 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.629602 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.629672 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.629765 (+ 93us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:55:31.630756 (+ 991us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.631650 (+ 894us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.633467 (+ 1817us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.634360 (+ 893us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.634420 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.634591 (+ 171us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.634739 (+ 148us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:55:31.635875 17696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89f02fcbe4c247ab9be0b84b0d18c2d0" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f2843631284d3a9d5dd82ca0107b6d" is_pre_election: true
I20251027 14:55:31.636094 17433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89f02fcbe4c247ab9be0b84b0d18c2d0" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdd8ffe904445fd96eee9be6fb96300" is_pre_election: true
I20251027 14:55:31.636338 17696 raft_consensus.cc:2468] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 0.
I20251027 14:55:31.636713 17433 raft_consensus.cc:2468] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 0.
I20251027 14:55:31.637238 17498 leader_election.cc:304] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d; no voters:
I20251027 14:55:31.637629 17763 raft_consensus.cc:2804] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:55:31.637795 17763 raft_consensus.cc:493] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:55:31.637897 17763 raft_consensus.cc:3060] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:31.640256 17763 raft_consensus.cc:515] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.640951 17763 leader_election.cc:290] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 election: Requested vote from peers 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585), 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255)
I20251027 14:55:31.641557 17696 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89f02fcbe4c247ab9be0b84b0d18c2d0" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "76f2843631284d3a9d5dd82ca0107b6d"
I20251027 14:55:31.641502 17433 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "89f02fcbe4c247ab9be0b84b0d18c2d0" candidate_uuid: "316eeb2c599f436bad9b6870f494993e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6cdd8ffe904445fd96eee9be6fb96300"
I20251027 14:55:31.641793 17696 raft_consensus.cc:3060] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:31.641857 17433 raft_consensus.cc:3060] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:55:31.644232 17696 raft_consensus.cc:2468] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 1.
I20251027 14:55:31.644351 17433 raft_consensus.cc:2468] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 316eeb2c599f436bad9b6870f494993e in term 1.
I20251027 14:55:31.644901 17498 leader_election.cc:304] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d; no voters:
I20251027 14:55:31.645258 17763 raft_consensus.cc:2804] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:55:31.645725 17763 raft_consensus.cc:697] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 1 LEADER]: Becoming Leader. State: Replica: 316eeb2c599f436bad9b6870f494993e, State: Running, Role: LEADER
I20251027 14:55:31.646253 17763 consensus_queue.cc:237] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:31.651429 17292 catalog_manager.cc:5649] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e reported cstate change: term changed from 0 to 1, leader changed from <none> to 316eeb2c599f436bad9b6870f494993e (127.0.104.2). New cstate: current_term: 1 leader_uuid: "316eeb2c599f436bad9b6870f494993e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } health_report { overall_health: HEALTHY } } }
W20251027 14:55:31.666968 17611 tablet.cc:2378] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:55:31.668850 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.662807 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53644 (local address 127.0.104.2:43391)
1027 14:55:31.662995 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.663001 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.663081 (+ 80us) server_negotiation.cc:408] Connection header received
1027 14:55:31.663308 (+ 227us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.663313 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.663396 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.663529 (+ 133us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:55:31.664103 (+ 574us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.665086 (+ 983us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.666531 (+ 1445us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.667047 (+ 516us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.667265 (+ 218us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:55:31.668042 (+ 777us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:55:31.668204 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.668358 (+ 154us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.668531 (+ 173us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:55:31.676877 17433 raft_consensus.cc:1275] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Refusing update from remote peer 316eeb2c599f436bad9b6870f494993e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:55:31.676875 17696 raft_consensus.cc:1275] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Refusing update from remote peer 316eeb2c599f436bad9b6870f494993e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:55:31.677810 17768 consensus_queue.cc:1048] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:55:31.678049 17763 consensus_queue.cc:1048] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [LEADER]: Connected to new peer: Peer: permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:55:31.689254 17770 mvcc.cc:204] Tried to move back new op lower bound from 7215419112137867264 to 7215419112027242496. Current Snapshot: MvccSnapshot[applied={T|T < 7215419112137867264}]
I20251027 14:55:31.690062 17771 mvcc.cc:204] Tried to move back new op lower bound from 7215419112137867264 to 7215419112027242496. Current Snapshot: MvccSnapshot[applied={T|T < 7215419112137867264}]
I20251027 14:55:31.704309 17292 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42638:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:55:31.704593 17292 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:42638:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:55:31.708218 17292 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Sending DeleteTablet for 3 replicas of tablet 89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:55:31.709530 17676 tablet_service.cc:1552] Processing DeleteTablet for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:55:31 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:51880
I20251027 14:55:31.709641 17413 tablet_service.cc:1552] Processing DeleteTablet for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:55:31 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:46534
I20251027 14:55:31.709656 17545 tablet_service.cc:1552] Processing DeleteTablet for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:55:31 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:53638
I20251027 14:55:31.710932 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 17329
I20251027 14:55:31.720420 17780 tablet_replica.cc:333] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: stopping tablet replica
I20251027 14:55:31.720906 17780 raft_consensus.cc:2243] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:55:31.721422 17780 raft_consensus.cc:2272] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:55:31.722220 17781 tablet_replica.cc:333] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: stopping tablet replica
I20251027 14:55:31.722641 17781 raft_consensus.cc:2243] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:55:31.723331 17781 raft_consensus.cc:2272] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:55:31.725894 17780 ts_tablet_manager.cc:1916] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:55:31.730628 17779 tablet_replica.cc:333] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: stopping tablet replica
I20251027 14:55:31.731060 17779 raft_consensus.cc:2243] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:55:31.731564 17779 raft_consensus.cc:2272] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:55:31.731683 17781 ts_tablet_manager.cc:1916] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:55:31.733115 17752 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.712610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46546 (local address 127.0.104.1:45585)
1027 14:55:31.712804 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.712810 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.712829 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:55:31.712896 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.712902 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.712985 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.713084 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:31.715587 (+ 2503us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.717164 (+ 1577us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.719672 (+ 2508us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.720249 (+ 577us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.724341 (+ 4092us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:31.724440 (+ 99us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:31.724461 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:31.724515 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:31.729087 (+ 4572us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.730263 (+ 1176us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.730273 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.730277 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.730382 (+ 105us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.730774 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.730780 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.730783 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.731110 (+ 327us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:31.731381 (+ 271us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.732489 (+ 1108us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.732753 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":81}
I20251027 14:55:31.733888 17473 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:31.734556 17779 ts_tablet_manager.cc:1916] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:55:31.735309 17780 ts_tablet_manager.cc:1929] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:55:31.735487 17780 log.cc:1199] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:55:31.736008 17780 ts_tablet_manager.cc:1950] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 76f2843631284d3a9d5dd82ca0107b6d: Deleting consensus metadata
I20251027 14:55:31.738023 17276 catalog_manager.cc:4985] TS 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255): tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 (table test-table [id=d97b042f8a2d4efdb5e990e57ff6dd55]) successfully deleted
I20251027 14:55:31.738857 17781 ts_tablet_manager.cc:1929] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:55:31.739010 17781 log.cc:1199] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:55:31.739426 17781 ts_tablet_manager.cc:1950] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 316eeb2c599f436bad9b6870f494993e: Deleting consensus metadata
I20251027 14:55:31.741080 17277 catalog_manager.cc:4985] TS 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391): tablet 89f02fcbe4c247ab9be0b84b0d18c2d0 (table test-table [id=d97b042f8a2d4efdb5e990e57ff6dd55]) successfully deleted
I20251027 14:55:31.827797 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 17329
I20251027 14:55:31.843096 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 17482
W20251027 14:55:31.843139 17279 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:45585 (error 108)
I20251027 14:55:31.846318 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.845833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46554)
1027 14:55:31.846016 (+ 183us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.846133 (+ 117us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
W20251027 14:55:31.846668 17279 catalog_manager.cc:4712] TS 6cdd8ffe904445fd96eee9be6fb96300 (127.0.104.1:45585): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 89f02fcbe4c247ab9be0b84b0d18c2d0: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
I20251027 14:55:31.857740 17756 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.844168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53658 (local address 127.0.104.2:43391)
1027 14:55:31.844335 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.844341 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.844362 (+ 21us) server_negotiation.cc:408] Connection header received
1027 14:55:31.844475 (+ 113us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.844482 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.844568 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.844684 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:31.846465 (+ 1781us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.847514 (+ 1049us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.848900 (+ 1386us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.849356 (+ 456us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.851247 (+ 1891us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:31.851290 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:31.851306 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:31.851349 (+ 43us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:31.854519 (+ 3170us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.855251 (+ 732us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.855263 (+ 12us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.855266 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.855376 (+ 110us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.856003 (+ 627us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.856007 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.856010 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.856343 (+ 333us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:31.856800 (+ 457us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.857126 (+ 326us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.857404 (+ 278us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:55:31.858500 17605 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:31.900137 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.899595 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46558)
1027 14:55:31.899826 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.899955 (+ 129us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20251027 14:55:31.901228 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.900854 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46566)
1027 14:55:31.901003 (+ 149us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.901084 (+ 81us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":45}
I20251027 14:55:31.961733 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.961169 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46578)
1027 14:55:31.961378 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.961510 (+ 132us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20251027 14:55:31.962759 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.962397 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46584)
1027 14:55:31.962539 (+ 142us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:31.962621 (+ 82us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251027 14:55:31.969229 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 17482
I20251027 14:55:31.985306 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 17613
I20251027 14:55:31.997926 17754 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:31.986070 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51884 (local address 127.0.104.3:36255)
1027 14:55:31.986257 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:31.986262 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:31.986282 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:55:31.986417 (+ 135us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:31.986422 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:31.986487 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:31.986580 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:31.988004 (+ 1424us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.988900 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:31.990073 (+ 1173us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:31.990513 (+ 440us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:31.992428 (+ 1915us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:31.992471 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:31.992488 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:31.992526 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:31.995336 (+ 2810us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.996029 (+ 693us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.996037 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.996039 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.996125 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:31.996446 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:31.996452 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:31.996455 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:31.996745 (+ 290us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:31.997055 (+ 310us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:31.997322 (+ 267us) server_negotiation.cc:300] Negotiation successful
1027 14:55:31.997560 (+ 238us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:55:31.998565 17736 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:32.043268 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:32.042769 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46596)
1027 14:55:32.042980 (+ 211us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:32.043091 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":76}
I20251027 14:55:32.044333 17753 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:32.043968 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45585 (local address 127.0.0.1:46598)
1027 14:55:32.044148 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:32.044218 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45585: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":99}
I20251027 14:55:32.087683 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 17613
I20251027 14:55:32.101204 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 17259
I20251027 14:55:32.102396 17322 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:55:32.215464 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 17259
I20251027 14:55:52.236835 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37463
--webserver_interface=127.0.104.62
--webserver_port=45481
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37463
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20 with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:52.567116 17787 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:52.567404 17787 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:52.567502 17787 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:52.575563 17787 flags.cc:425] Enabled experimental flag: --authn_token_validity_seconds=20
W20251027 14:55:52.575656 17787 flags.cc:425] Enabled experimental flag: --authz_token_validity_seconds=20
W20251027 14:55:52.575704 17787 flags.cc:425] Enabled experimental flag: --tsk_rotation_seconds=20
W20251027 14:55:52.575747 17787 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:55:52.575814 17787 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:52.575865 17787 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:52.575947 17787 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:55:52.576041 17787 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:55:52.586360 17787 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
--authn_token_validity_seconds=20
--authz_token_validity_seconds=20
--tsk_rotation_seconds=20
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37463
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37463
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=45481
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:52.587966 17787 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:52.589360 17787 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:52.598290 17792 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:52.598351 17795 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:52.598254 17793 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:52.598594 17787 server_base.cc:1047] running on GCE node
I20251027 14:55:52.599750 17787 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:52.601282 17787 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:52.602509 17787 hybrid_clock.cc:648] HybridClock initialized: now 1761576952602454 us; error 37 us; skew 500 ppm
I20251027 14:55:52.602864 17787 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:52 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:52.606426 17787 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:55:52.608175 17787 webserver.cc:492] Webserver started at http://127.0.104.62:45481/ using document root <none> and password file <none>
I20251027 14:55:52.609017 17787 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:52.609131 17787 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:52.615114 17787 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20251027 14:55:52.618315 17802 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:52.619746 17787 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:55:52.619925 17787 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "9739ec864cad41c2a9f68621e45e22b6"
format_stamp: "Formatted at 2025-10-27 14:55:06 on dist-test-slave-hxxt"
server_key: "0970753bc6e509e10558557031182cc0"
server_key_iv: "5775589f458c332f44348c6abc249df8"
server_key_version: "encryptionkey@0"
I20251027 14:55:52.620728 17787 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:52.634275 17787 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:52.638561 17787 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:52.638819 17787 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:52.659060 17787 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37463
I20251027 14:55:52.659076 17853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37463 every 8 connection(s)
I20251027 14:55:52.660753 17787 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:55:52.667817 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17787
I20251027 14:55:52.667894 17854 sys_catalog.cc:263] Verifying existing consensus state
I20251027 14:55:52.668689 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:45585
--local_ip_for_outbound_sockets=127.0.104.1
--tserver_master_addrs=127.0.104.62:37463
--webserver_port=41283
--webserver_interface=127.0.104.1
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:55:52.671260 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap starting.
I20251027 14:55:52.690097 17854 log.cc:826] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:52.706094 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap replayed 1/1 log segments. Stats: ops{read=14 overwritten=0 applied=14 ignored=0} inserts{seen=8 ignored=0} mutations{seen=8 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:55:52.706568 17854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Bootstrap complete.
I20251027 14:55:52.713001 17854 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:52.713675 17854 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9739ec864cad41c2a9f68621e45e22b6, State: Initialized, Role: FOLLOWER
I20251027 14:55:52.714509 17854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:52.714730 17854 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:55:52.714839 17854 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:55:52.715015 17854 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 2 FOLLOWER]: Advancing to term 3
I20251027 14:55:52.718698 17854 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:52.719203 17854 leader_election.cc:304] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9739ec864cad41c2a9f68621e45e22b6; no voters:
I20251027 14:55:52.719789 17854 leader_election.cc:290] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [CANDIDATE]: Term 3 election: Requested vote from peers
I20251027 14:55:52.719930 17858 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 3 FOLLOWER]: Leader election won for term 3
I20251027 14:55:52.721042 17858 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [term 3 LEADER]: Becoming Leader. State: Replica: 9739ec864cad41c2a9f68621e45e22b6, State: Running, Role: LEADER
I20251027 14:55:52.721625 17858 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 14, Committed index: 14, Last appended: 2.14, Last appended by leader: 14, Current term: 3, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } }
I20251027 14:55:52.722054 17854 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:55:52.724017 17860 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9739ec864cad41c2a9f68621e45e22b6. Latest consensus state: current_term: 3 leader_uuid: "9739ec864cad41c2a9f68621e45e22b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } } }
I20251027 14:55:52.724397 17860 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: This master's current role is: LEADER
I20251027 14:55:52.724813 17859 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 leader_uuid: "9739ec864cad41c2a9f68621e45e22b6" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9739ec864cad41c2a9f68621e45e22b6" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37463 } } }
I20251027 14:55:52.725086 17859 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6 [sys.catalog]: This master's current role is: LEADER
I20251027 14:55:52.727730 17866 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:55:52.736701 17866 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:55:52.738317 17866 catalog_manager.cc:1269] Loaded cluster ID: dd4315ed6b8d4f1f8488384b540ab6b6
I20251027 14:55:52.738488 17866 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:55:52.743093 17866 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:55:52.745203 17866 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Loaded TSK: 0
I20251027 14:55:52.745338 17875 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251027 14:55:52.745512 17866 catalog_manager.cc:6033] T 00000000000000000000000000000000 P 9739ec864cad41c2a9f68621e45e22b6: Loaded TSK: 1
F20251027 14:55:52.746235 17866 token_signing_key.cc:66] Check failed: _s.ok() Bad status: Runtime error: unable to load data from memory: error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=RSAPrivateKey error:04093004:rsa routines:old_rsa_priv_decode:RSA lib:../crypto/rsa/rsa_ameth.c:130 error:0D0680A8:asn1 encoding routines:asn1_check_tlen:wrong tag:../crypto/asn1/tasn_dec.c:1149 error:0D06C03A:asn1 encoding routines:asn1_d2i_ex_primitive:nested asn1 error:../crypto/asn1/tasn_dec.c:713 error:0D08303A:asn1 encoding routines:asn1_template_noexp_d2i:nested asn1 error:../crypto/asn1/tasn_dec.c:646:Field=version, Type=PKCS8_PRIV_KEY_INFO
*** Check failure stack trace: ***
@ 0x7fb07a6bfdcd google::LogMessage::Fail() at ??:0
@ 0x7fb07a6c3b93 google::LogMessage::SendToLog() at ??:0
@ 0x7fb07a6bf7cc google::LogMessage::Flush() at ??:0
@ 0x7fb07a6c0f59 google::LogMessageFatal::~LogMessageFatal() at ??:0
@ 0x7fb07e657b64 kudu::security::TokenSigningPrivateKey::TokenSigningPrivateKey() at ??:0
@ 0x7fb07e646b6e kudu::security::TokenSigner::ImportKeys() at ??:0
@ 0x7fb090eff36a kudu::master::CatalogManager::LoadTskEntries() at ??:0
@ 0x7fb090efe913 kudu::master::CatalogManager::InitTokenSigner() at ??:0
@ 0x7fb090f93657 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_10::operator()() at ??:0
@ 0x7fb090f93474 std::_Function_handler<>::_M_invoke() at ??:0
@ 0x7fb090fde789 std::function<>::operator()() at ??:0
@ 0x7fb090f02f75 kudu::master::CatalogManager::PrepareForLeadershipTask()::$_6::operator()() at ??:0
@ 0x7fb090f00e46 kudu::master::CatalogManager::PrepareForLeadershipTask() at ??:0
@ 0x7fb090f919c0 kudu::master::CatalogManager::ElectedAsLeaderCb()::$_2::operator()() at ??:0
@ 0x7fb090f917e9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7fb07d454d80 kudu::ThreadPool::DispatchThread() at ??:0
@ 0x7fb07d45804c kudu::ThreadPool::CreateThread()::$_1::operator()() at ??:0
@ 0x7fb07d457ec9 std::_Function_handler<>::_M_invoke() at ??:0
@ 0xa8867d std::function<>::operator()() at ??:?
@ 0x7fb07d433ee9 kudu::Thread::SuperviseThread() at ??:0
@ 0x7fb080a3b6db start_thread at ??:0
@ 0x7fb07957071f clone at ??:0
W20251027 14:55:53.053920 17856 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:53.054229 17856 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:53.054333 17856 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:53.062434 17856 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:53.062541 17856 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:53.062701 17856 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:55:53.073105 17856 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:45585
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=41283
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:53.074602 17856 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:53.075906 17856 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:53.084794 17901 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:53.084858 17902 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:53.084834 17904 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:53.085920 17856 server_base.cc:1047] running on GCE node
I20251027 14:55:53.086634 17856 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:53.087605 17856 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:53.088886 17856 hybrid_clock.cc:648] HybridClock initialized: now 1761576953088818 us; error 34 us; skew 500 ppm
I20251027 14:55:53.089218 17856 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:53 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:53.092504 17856 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:55:53.094080 17856 webserver.cc:492] Webserver started at http://127.0.104.1:41283/ using document root <none> and password file <none>
I20251027 14:55:53.094853 17856 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:53.094954 17856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:53.100735 17856 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20251027 14:55:53.103776 17911 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:53.105223 17856 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:55:53.105403 17856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "6cdd8ffe904445fd96eee9be6fb96300"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "8ebf221173297bc71cc58e6ec705eb33"
server_key_iv: "342966c7e492a522770e809d81c04d6f"
server_key_version: "encryptionkey@0"
I20251027 14:55:53.106143 17856 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:53.121232 17856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:53.125422 17856 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:53.125698 17856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:53.126705 17856 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:53.128662 17918 ts_tablet_manager.cc:548] Loading tablet metadata (0/1 complete)
I20251027 14:55:53.134891 17856 ts_tablet_manager.cc:585] Loaded tablet metadata (1 total tablets, 1 live tablets)
I20251027 14:55:53.134992 17856 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.007s user 0.000s sys 0.001s
I20251027 14:55:53.135121 17856 ts_tablet_manager.cc:600] Registering tablets (0/1 complete)
I20251027 14:55:53.138139 17856 ts_tablet_manager.cc:616] Registered 1 tablets
I20251027 14:55:53.138247 17856 ts_tablet_manager.cc:595] Time spent register tablets: real 0.003s user 0.000s sys 0.001s
I20251027 14:55:53.138728 17918 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Bootstrap starting.
I20251027 14:55:53.147377 17918 log.cc:826] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Log is configured to *not* fsync() on all Append() calls
I20251027 14:55:53.161353 17918 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Bootstrap replayed 1/1 log segments. Stats: ops{read=2 overwritten=0 applied=2 ignored=0} inserts{seen=1 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20251027 14:55:53.162103 17918 tablet_bootstrap.cc:492] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Bootstrap complete.
I20251027 14:55:53.162678 17918 ts_tablet_manager.cc:1403] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Time spent bootstrapping tablet: real 0.024s user 0.018s sys 0.005s
I20251027 14:55:53.168998 17918 raft_consensus.cc:359] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:53.169894 17918 raft_consensus.cc:740] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6cdd8ffe904445fd96eee9be6fb96300, State: Initialized, Role: FOLLOWER
I20251027 14:55:53.170773 17918 consensus_queue.cc:260] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 2, Last appended: 1.2, Last appended by leader: 2, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:53.172299 17918 ts_tablet_manager.cc:1434] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300: Time spent starting tablet: real 0.009s user 0.005s sys 0.004s
I20251027 14:55:53.174546 17856 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45585
I20251027 14:55:53.174611 18026 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45585 every 8 connection(s)
I20251027 14:55:53.175776 17856 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:55:53.180339 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:53.178514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:37539)
1027 14:55:53.179298 (+ 784us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:53.179481 (+ 183us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":553,"thread_start_us":315,"threads_started":1}
W20251027 14:55:53.180905 18027 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:37463 (0 consecutive failures): Network error: Failed to ping master at 127.0.104.62:37463: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
I20251027 14:55:53.181710 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:53.181382 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:49299)
1027 14:55:53.181522 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:53.181597 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251027 14:55:53.181751 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 17856
I20251027 14:55:53.182379 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:43391
--local_ip_for_outbound_sockets=127.0.104.2
--tserver_master_addrs=127.0.104.62:37463
--webserver_port=35089
--webserver_interface=127.0.104.2
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:55:53.182510 18029 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:53.182197 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:52273)
1027 14:55:53.182343 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:53.182397 (+ 54us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251027 14:55:53.182716 18027 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
W20251027 14:55:53.500146 18030 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:53.500468 18030 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:53.500617 18030 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:53.508823 18030 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:53.508919 18030 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:53.509121 18030 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:55:53.519588 18030 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:43391
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=35089
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:53.521150 18030 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:53.522429 18030 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:53.531250 18036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:53.531378 18035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:53.531890 18038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:53.532806 18030 server_base.cc:1047] running on GCE node
I20251027 14:55:53.533406 18030 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:53.534271 18030 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:53.535513 18030 hybrid_clock.cc:648] HybridClock initialized: now 1761576953535442 us; error 39 us; skew 500 ppm
I20251027 14:55:53.535799 18030 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:53 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:53.538832 18030 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:55:53.540261 18030 webserver.cc:492] Webserver started at http://127.0.104.2:35089/ using document root <none> and password file <none>
I20251027 14:55:53.540988 18030 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:53.541090 18030 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:53.546408 18030 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251027 14:55:53.549038 18045 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:53.550261 18030 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:55:53.550418 18030 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "316eeb2c599f436bad9b6870f494993e"
format_stamp: "Formatted at 2025-10-27 14:55:07 on dist-test-slave-hxxt"
server_key: "c3d244e9494ed6c2fd4854efe5012056"
server_key_iv: "074aebb9b52498e220e5ce11667f8ab3"
server_key_version: "encryptionkey@0"
I20251027 14:55:53.551059 18030 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:53.573546 18030 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:53.580782 18030 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:53.581071 18030 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:53.582051 18030 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:53.583559 18030 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:53.583634 18030 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:53.583738 18030 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:53.583788 18030 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:53.615574 18030 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:43391
I20251027 14:55:53.615621 18157 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:43391 every 8 connection(s)
I20251027 14:55:53.617012 18030 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:55:53.622126 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:53.620276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:35765)
1027 14:55:53.621047 (+ 771us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:53.621231 (+ 184us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":519,"thread_start_us":320,"threads_started":1}
W20251027 14:55:53.622748 18158 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:37463 (0 consecutive failures): Network error: Failed to ping master at 127.0.104.62:37463: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
I20251027 14:55:53.623574 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:53.623218 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:50429)
1027 14:55:53.623384 (+ 166us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:53.623463 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
I20251027 14:55:53.624366 18160 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:53.624074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:46279)
1027 14:55:53.624178 (+ 104us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:53.624233 (+ 55us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":35}
W20251027 14:55:53.624580 18158 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251027 14:55:53.624699 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18030
I20251027 14:55:53.625281 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:36255
--local_ip_for_outbound_sockets=127.0.104.3
--tserver_master_addrs=127.0.104.62:37463
--webserver_port=45159
--webserver_interface=127.0.104.3
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43735
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:55:53.942128 18161 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:55:53.942405 18161 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:55:53.942508 18161 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:55:53.950456 18161 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:55:53.950554 18161 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:55:53.950711 18161 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:55:53.963833 18161 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43735
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:36255
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=45159
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:37463
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:55:53.965646 18161 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:55:53.966882 18161 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:55:53.975615 18169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:53.975641 18167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:55:53.975804 18166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:55:53.976473 18161 server_base.cc:1047] running on GCE node
I20251027 14:55:53.977001 18161 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:55:53.977835 18161 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:55:53.979061 18161 hybrid_clock.cc:648] HybridClock initialized: now 1761576953978984 us; error 44 us; skew 500 ppm
I20251027 14:55:53.979360 18161 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:55:53 dist-test-slave-hxxt krb5kdc[16695](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:55:53.982455 18161 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:55:53.983944 18161 webserver.cc:492] Webserver started at http://127.0.104.3:45159/ using document root <none> and password file <none>
I20251027 14:55:53.984750 18161 fs_manager.cc:362] Metadata directory not provided
I20251027 14:55:53.984835 18161 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:55:53.990191 18161 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20251027 14:55:53.992940 18176 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:53.994230 18161 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:55:53.994385 18161 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "76f2843631284d3a9d5dd82ca0107b6d"
format_stamp: "Formatted at 2025-10-27 14:55:08 on dist-test-slave-hxxt"
server_key: "dbb94d1af6f3ebde789c35d378e7fcba"
server_key_iv: "3af22ea873c95260588ef53417dfadbf"
server_key_version: "encryptionkey@0"
I20251027 14:55:53.995066 18161 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:55:54.019520 18161 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:55:54.023355 18161 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:55:54.023648 18161 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:55:54.024600 18161 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:55:54.026049 18161 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:55:54.026127 18161 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:54.026237 18161 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:55:54.026294 18161 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:55:54.056854 18161 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36255
I20251027 14:55:54.056896 18288 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36255 every 8 connection(s)
I20251027 14:55:54.058270 18161 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.SecurityITest.TestEnableAndDisableEncryptedTSK.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:55:54.063032 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.061297 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:38071)
1027 14:55:54.062011 (+ 714us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.062203 (+ 192us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":483,"thread_start_us":305,"threads_started":1}
W20251027 14:55:54.063620 18289 heartbeater.cc:646] Failed to heartbeat to 127.0.104.62:37463 (0 consecutive failures): Network error: Failed to ping master at 127.0.104.62:37463: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
I20251027 14:55:54.064316 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.064027 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:42831)
1027 14:55:54.064135 (+ 108us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.064219 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":34}
I20251027 14:55:54.064982 18291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.064742 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:39881)
1027 14:55:54.064833 (+ 91us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.064885 (+ 52us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":33}
W20251027 14:55:54.065179 18289 heartbeater.cc:412] Failed 3 heartbeats in a row: no longer allowing fast heartbeat attempts.
I20251027 14:55:54.065801 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18161
I20251027 14:55:54.183918 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.183215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:44967)
1027 14:55:54.183651 (+ 436us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.183727 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":341,"thread_start_us":230,"threads_started":1}
I20251027 14:55:54.323351 18294 raft_consensus.cc:493] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:55:54.323640 18294 raft_consensus.cc:515] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:54.325330 18294 leader_election.cc:290] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255), 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391)
Oct 27 14:55:54 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
Oct 27 14:55:54 dist-test-slave-hxxt krb5kdc[16695](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576953, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:55:54.345263 18295 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.325768 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36255 (local address 127.0.104.1:53699)
1027 14:55:54.327888 (+ 2120us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.327905 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:54.328041 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:54.330210 (+ 2169us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:54.330222 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:54.331174 (+ 952us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:54.332076 (+ 902us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:54.332097 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.333668 (+ 1571us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:54.333672 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:54.334503 (+ 831us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:54.334514 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.334707 (+ 193us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:54.335590 (+ 883us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:54.335627 (+ 37us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:54.338792 (+ 3165us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:54.342219 (+ 3427us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:54.342224 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:54.342228 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:54.342641 (+ 413us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:54.342989 (+ 348us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:54.342993 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:54.342995 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:54.343085 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:54.343764 (+ 679us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:54.343772 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:54.344016 (+ 244us) client_negotiation.cc:769] Sending connection context
1027 14:55:54.344351 (+ 335us) client_negotiation.cc:240] Negotiation successful
1027 14:55:54.344838 (+ 487us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":2012,"spinlock_wait_cycles":42752,"thread_start_us":1881,"threads_started":1}
I20251027 14:55:54.345333 18297 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.325962 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:53699 (local address 127.0.104.3:36255)
1027 14:55:54.327672 (+ 1710us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:54.327689 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:54.328022 (+ 333us) server_negotiation.cc:408] Connection header received
1027 14:55:54.329312 (+ 1290us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:54.329321 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:54.329735 (+ 414us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:54.330079 (+ 344us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:54.332261 (+ 2182us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.333530 (+ 1269us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:54.334628 (+ 1098us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.335162 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:54.338969 (+ 3807us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:54.339045 (+ 76us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:54.339059 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:54.339093 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:54.342056 (+ 2963us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:54.342748 (+ 692us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:54.342756 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:54.342764 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:54.342878 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:54.343181 (+ 303us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:54.343184 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:54.343186 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:54.343605 (+ 419us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:54.343768 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:54.344615 (+ 847us) server_negotiation.cc:300] Negotiation successful
1027 14:55:54.344984 (+ 369us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1548,"thread_start_us":1401,"threads_started":1}
I20251027 14:55:54.345377 18293 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.325730 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:43391 (local address 127.0.104.1:41567)
1027 14:55:54.326094 (+ 364us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.326131 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:55:54.326816 (+ 685us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:55:54.330210 (+ 3394us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:55:54.330222 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:55:54.331194 (+ 972us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:55:54.332077 (+ 883us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:54.332097 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.333639 (+ 1542us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:55:54.333643 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:55:54.334462 (+ 819us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:55:54.334473 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.334760 (+ 287us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:54.335810 (+ 1050us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:55:54.335829 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:55:54.338441 (+ 2612us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:55:54.341956 (+ 3515us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:54.341966 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:54.341982 (+ 16us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:54.342554 (+ 572us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:54.342926 (+ 372us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:55:54.342930 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:55:54.342932 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:55:54.343040 (+ 108us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:55:54.343766 (+ 726us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:55:54.343774 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:55:54.344016 (+ 242us) client_negotiation.cc:769] Sending connection context
1027 14:55:54.344357 (+ 341us) client_negotiation.cc:240] Negotiation successful
1027 14:55:54.344985 (+ 628us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":256,"mutex_wait_us":184,"spinlock_wait_cycles":27136}
I20251027 14:55:54.346163 18296 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.325870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:41567 (local address 127.0.104.2:43391)
1027 14:55:54.328105 (+ 2235us) server_negotiation.cc:207] Beginning negotiation
1027 14:55:54.328120 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:55:54.328161 (+ 41us) server_negotiation.cc:408] Connection header received
1027 14:55:54.329302 (+ 1141us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:55:54.329312 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:55:54.329721 (+ 409us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:55:54.330073 (+ 352us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:55:54.332271 (+ 2198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.333474 (+ 1203us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:55:54.334588 (+ 1114us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:55:54.335119 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:55:54.338664 (+ 3545us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:55:54.338766 (+ 102us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:55:54.338786 (+ 20us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:55:54.338827 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:55:54.341767 (+ 2940us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:54.342677 (+ 910us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:54.342685 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:54.342691 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:54.342792 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:55:54.343145 (+ 353us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:55:54.343150 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:55:54.343152 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:55:54.343618 (+ 466us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:55:54.343777 (+ 159us) server_negotiation.cc:1036] Waiting for connection context
1027 14:55:54.344626 (+ 849us) server_negotiation.cc:300] Negotiation successful
1027 14:55:54.344977 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2083,"thread_start_us":1932,"threads_started":1}
W20251027 14:55:54.348855 17912 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
W20251027 14:55:54.349251 17913 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:55:54.349393 17913 leader_election.cc:304] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cdd8ffe904445fd96eee9be6fb96300; no voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d
I20251027 14:55:54.349764 18294 raft_consensus.cc:2749] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:55:54.625882 18298 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:54.625078 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:60333)
1027 14:55:54.625517 (+ 439us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:54.625636 (+ 119us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":221,"threads_started":1}
I20251027 14:55:55.066360 18299 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:55.065653 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:58655)
1027 14:55:55.066075 (+ 422us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:55.066170 (+ 95us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":316,"thread_start_us":203,"threads_started":1}
I20251027 14:55:55.185570 18300 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:55.184762 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:59489)
1027 14:55:55.185269 (+ 507us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:55.185356 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":356,"thread_start_us":228,"threads_started":1}
I20251027 14:55:55.627385 18301 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:55.626681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:39755)
1027 14:55:55.627113 (+ 432us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:55.627187 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":344,"thread_start_us":235,"threads_started":1}
I20251027 14:55:56.067979 18302 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:56.067217 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:35443)
1027 14:55:56.067741 (+ 524us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:56.067818 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":437,"thread_start_us":262,"threads_started":1}
I20251027 14:55:56.187198 18303 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:56.186458 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:43471)
1027 14:55:56.186925 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:56.187019 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":369,"thread_start_us":264,"threads_started":1}
I20251027 14:55:56.524578 18304 raft_consensus.cc:493] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:55:56.524778 18304 raft_consensus.cc:515] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:56.525586 18304 leader_election.cc:290] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255), 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391)
W20251027 14:55:56.526484 17912 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
W20251027 14:55:56.526669 17913 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:55:56.526772 17913 leader_election.cc:304] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cdd8ffe904445fd96eee9be6fb96300; no voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d
I20251027 14:55:56.527012 18304 raft_consensus.cc:2749] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:55:56.628998 18305 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:56.628295 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:36439)
1027 14:55:56.628760 (+ 465us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:56.628838 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":267,"threads_started":1}
I20251027 14:55:57.069428 18306 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:57.068786 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:39427)
1027 14:55:57.069196 (+ 410us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:57.069271 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":313,"thread_start_us":208,"threads_started":1}
I20251027 14:55:57.188895 18307 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:57.188132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:40565)
1027 14:55:57.188612 (+ 480us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:57.188696 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":260,"threads_started":1}
I20251027 14:55:57.630724 18308 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:57.629882 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:58997)
1027 14:55:57.630364 (+ 482us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:57.630465 (+ 101us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":361,"thread_start_us":233,"threads_started":1}
I20251027 14:55:58.070977 18309 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:58.070241 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:32777)
1027 14:55:58.070711 (+ 470us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:58.070786 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":362,"thread_start_us":243,"threads_started":1}
I20251027 14:55:58.190589 18310 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:58.189856 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:55147)
1027 14:55:58.190324 (+ 468us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:58.190410 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":214,"threads_started":1}
I20251027 14:55:58.632550 18311 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:58.631737 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:58677)
1027 14:55:58.632256 (+ 519us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:58.632342 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":413,"thread_start_us":287,"threads_started":1}
I20251027 14:55:59.072579 18312 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:59.071841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:46203)
1027 14:55:59.072312 (+ 471us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:59.072419 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":371,"thread_start_us":253,"threads_started":1}
I20251027 14:55:59.161172 18313 raft_consensus.cc:493] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:55:59.161368 18313 raft_consensus.cc:515] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:55:59.162019 18313 leader_election.cc:290] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255), 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391)
W20251027 14:55:59.162909 17913 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
W20251027 14:55:59.163110 17912 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:55:59.163223 17912 leader_election.cc:304] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cdd8ffe904445fd96eee9be6fb96300; no voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d
I20251027 14:55:59.163431 18313 raft_consensus.cc:2749] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:55:59.192224 18314 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:59.191507 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:52049)
1027 14:55:59.191971 (+ 464us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:59.192046 (+ 75us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":372,"thread_start_us":274,"threads_started":1}
I20251027 14:55:59.634171 18315 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:55:59.633466 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:45457)
1027 14:55:59.633893 (+ 427us) negotiation.cc:107] Waiting for socket to connect
1027 14:55:59.633977 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":325,"thread_start_us":220,"threads_started":1}
I20251027 14:56:00.074215 18316 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:00.073486 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:59831)
1027 14:56:00.073953 (+ 467us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:00.074037 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":238,"threads_started":1}
I20251027 14:56:00.193779 18317 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:00.193085 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:52237)
1027 14:56:00.193509 (+ 424us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:00.193585 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":327,"thread_start_us":220,"threads_started":1}
I20251027 14:56:00.635948 18318 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:00.635091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:50527)
1027 14:56:00.635658 (+ 567us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:00.635743 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":449,"thread_start_us":252,"threads_started":1}
I20251027 14:56:01.075927 18319 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:01.075112 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:44695)
1027 14:56:01.075623 (+ 511us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:01.075717 (+ 94us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":417,"thread_start_us":253,"threads_started":1}
I20251027 14:56:01.195277 18320 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:01.194622 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:53301)
1027 14:56:01.195016 (+ 394us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:01.195108 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":303,"thread_start_us":205,"threads_started":1}
I20251027 14:56:01.637647 18321 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:01.636943 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:50871)
1027 14:56:01.637388 (+ 445us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:01.637466 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":239,"threads_started":1}
I20251027 14:56:02.077677 18322 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:02.076923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:60645)
1027 14:56:02.077437 (+ 514us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:02.077509 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":421,"thread_start_us":244,"threads_started":1}
I20251027 14:56:02.196992 18323 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:02.196153 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:37095)
1027 14:56:02.196695 (+ 542us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:02.196780 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":440,"thread_start_us":322,"threads_started":1}
I20251027 14:56:02.639447 18324 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:02.638604 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:42777)
1027 14:56:02.639133 (+ 529us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:02.639218 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":415,"thread_start_us":268,"threads_started":1}
I20251027 14:56:03.079314 18325 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:03.078627 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:58653)
1027 14:56:03.079069 (+ 442us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:03.079153 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":235,"threads_started":1}
I20251027 14:56:03.198799 18326 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:03.198086 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:39479)
1027 14:56:03.198535 (+ 449us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:03.198621 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":350,"thread_start_us":213,"threads_started":1}
I20251027 14:56:03.428010 18327 raft_consensus.cc:493] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:03.428213 18327 raft_consensus.cc:515] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "6cdd8ffe904445fd96eee9be6fb96300" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45585 } } peers { permanent_uuid: "76f2843631284d3a9d5dd82ca0107b6d" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36255 } } peers { permanent_uuid: "316eeb2c599f436bad9b6870f494993e" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 43391 } }
I20251027 14:56:03.428884 18327 leader_election.cc:290] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255), 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391)
W20251027 14:56:03.429708 17912 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 76f2843631284d3a9d5dd82ca0107b6d (127.0.104.3:36255): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
W20251027 14:56:03.429863 17913 leader_election.cc:343] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Tablet error from VoteRequest() call to peer 316eeb2c599f436bad9b6870f494993e (127.0.104.2:43391): Not found: Tablet not found: 89f02fcbe4c247ab9be0b84b0d18c2d0
I20251027 14:56:03.429998 17913 leader_election.cc:304] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6cdd8ffe904445fd96eee9be6fb96300; no voters: 316eeb2c599f436bad9b6870f494993e, 76f2843631284d3a9d5dd82ca0107b6d
I20251027 14:56:03.430239 18327 raft_consensus.cc:2749] T 89f02fcbe4c247ab9be0b84b0d18c2d0 P 6cdd8ffe904445fd96eee9be6fb96300 [term 1 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20251027 14:56:03.641307 18328 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:03.640527 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:55793)
1027 14:56:03.641018 (+ 491us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:03.641110 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":383,"thread_start_us":254,"threads_started":1}
I20251027 14:56:04.081213 18329 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:04.080465 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:57663)
1027 14:56:04.080942 (+ 477us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:04.081029 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":345,"thread_start_us":222,"threads_started":1}
I20251027 14:56:04.200553 18330 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:04.199790 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:58571)
1027 14:56:04.200262 (+ 472us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:04.200353 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":353,"thread_start_us":240,"threads_started":1}
I20251027 14:56:04.643004 18331 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:04.642263 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:52877)
1027 14:56:04.642729 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:04.642817 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":366,"thread_start_us":250,"threads_started":1}
I20251027 14:56:05.082770 18332 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:05.082064 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:33031)
1027 14:56:05.082518 (+ 454us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:05.082602 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":348,"thread_start_us":214,"threads_started":1}
I20251027 14:56:05.202189 18333 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:05.201457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:44131)
1027 14:56:05.201938 (+ 481us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:05.202026 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":363,"thread_start_us":246,"threads_started":1}
I20251027 14:56:05.644732 18334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:05.643946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:53781)
1027 14:56:05.644423 (+ 477us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:05.644511 (+ 88us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":347,"thread_start_us":224,"threads_started":1}
I20251027 14:56:06.084565 18335 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:06.083735 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:36841)
1027 14:56:06.084240 (+ 505us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:06.084326 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":290,"threads_started":1}
I20251027 14:56:06.203936 18336 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:06.203167 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:57745)
1027 14:56:06.203628 (+ 461us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:06.203735 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":354,"thread_start_us":227,"threads_started":1}
I20251027 14:56:06.646529 18337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:06.645738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:50739)
1027 14:56:06.646247 (+ 509us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:06.646363 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":412,"thread_start_us":236,"threads_started":1}
I20251027 14:56:07.086182 18338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:07.085457 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:42071)
1027 14:56:07.085915 (+ 458us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:07.086023 (+ 108us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":359,"thread_start_us":223,"threads_started":1}
I20251027 14:56:07.205579 18339 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:07.204861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:47817)
1027 14:56:07.205282 (+ 421us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:07.205362 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":322,"thread_start_us":217,"threads_started":1}
I20251027 14:56:07.648195 18340 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:07.647453 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:35039)
1027 14:56:07.647919 (+ 466us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:07.647998 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":367,"thread_start_us":254,"threads_started":1}
I20251027 14:56:08.087836 18341 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:08.087134 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:58417)
1027 14:56:08.087565 (+ 431us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:08.087639 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":339,"thread_start_us":239,"threads_started":1}
I20251027 14:56:08.207508 18342 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:08.206738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.1:50279)
1027 14:56:08.207241 (+ 503us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:08.207318 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":401,"thread_start_us":254,"threads_started":1}
I20251027 14:56:08.650177 18343 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:08.649431 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.2:34703)
1027 14:56:08.649925 (+ 494us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:08.650001 (+ 76us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":388,"thread_start_us":250,"threads_started":1}
I20251027 14:56:09.072700 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 17856
I20251027 14:56:09.086683 18344 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:09.073799 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55052 (local address 127.0.104.1:45585)
1027 14:56:09.074516 (+ 717us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:09.074527 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:09.074558 (+ 31us) server_negotiation.cc:408] Connection header received
1027 14:56:09.074637 (+ 79us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:09.074644 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:09.074935 (+ 291us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:09.075145 (+ 210us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:09.076825 (+ 1680us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:09.077920 (+ 1095us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:09.079087 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:09.079546 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:09.081494 (+ 1948us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:09.081548 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:09.081563 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:09.081599 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:09.084352 (+ 2753us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:09.084987 (+ 635us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:09.084994 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:09.084996 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:09.085078 (+ 82us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:09.085340 (+ 262us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:09.085344 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:09.085345 (+ 1us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:09.085641 (+ 296us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:09.085870 (+ 229us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:09.086102 (+ 232us) server_negotiation.cc:300] Negotiation successful
1027 14:56:09.086354 (+ 252us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":534,"thread_start_us":411,"threads_started":1}
I20251027 14:56:09.088019 18022 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:09.089742 18346 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:09.088927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:37463 (local address 127.0.104.3:39291)
1027 14:56:09.089439 (+ 512us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:09.089543 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.62:37463: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":376,"thread_start_us":205,"threads_started":1}
I20251027 14:56:09.177449 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 17856
I20251027 14:56:09.191847 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18030
I20251027 14:56:09.204667 18347 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:09.192536 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36474 (local address 127.0.104.2:43391)
1027 14:56:09.193001 (+ 465us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:09.193006 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:09.193022 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:56:09.193078 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:09.193083 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:09.193154 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:09.193253 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:09.194867 (+ 1614us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:09.195816 (+ 949us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:09.197066 (+ 1250us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:09.197488 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:09.199309 (+ 1821us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:09.199358 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:09.199362 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:09.199394 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:09.202056 (+ 2662us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:09.202781 (+ 725us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:09.202786 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:09.202789 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:09.202890 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:09.203194 (+ 304us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:09.203199 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:09.203202 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:09.203435 (+ 233us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:09.203694 (+ 259us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:09.203951 (+ 257us) server_negotiation.cc:300] Negotiation successful
1027 14:56:09.204232 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":371,"thread_start_us":234,"threads_started":1}
I20251027 14:56:09.205425 18153 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:09.290544 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18030
I20251027 14:56:09.305330 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18161
I20251027 14:56:09.318403 18349 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:09.306168 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45634 (local address 127.0.104.3:36255)
1027 14:56:09.306740 (+ 572us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:09.306745 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:09.306761 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:56:09.306816 (+ 55us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:09.306821 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:09.306888 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:09.306988 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:09.308695 (+ 1707us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:09.309716 (+ 1021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:09.310865 (+ 1149us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:09.311297 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:09.313294 (+ 1997us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:09.313336 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:09.313340 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:09.313380 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:09.315888 (+ 2508us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:09.316579 (+ 691us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:09.316584 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:09.316586 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:09.316667 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:09.316979 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:09.316983 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:09.316985 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:09.317242 (+ 257us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:09.317506 (+ 264us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:09.317823 (+ 317us) server_negotiation.cc:300] Negotiation successful
1027 14:56:09.318069 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":474,"thread_start_us":345,"threads_started":1}
I20251027 14:56:09.319165 18284 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:09.403496 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18161
2025-10-27T14:56:09Z chronyd exiting
[ OK ] SecurityITest.TestEnableAndDisableEncryptedTSK (65210 ms)
[----------] 31 tests from SecurityITest (335640 ms total)
[----------] 6 tests from EncryptionPolicyTest
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/0
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:09 dist-test-slave-hxxt krb5kdc[18353](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:09 dist-test-slave-hxxt krb5kdc[18353](info): set up 2 sockets
Oct 27 14:56:09 dist-test-slave-hxxt krb5kdc[18353](info): commencing operation
krb5kdc: starting...
W20251027 14:56:11.300554 416 mini_kdc.cc:121] Time spent starting KDC: real 1.856s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:11 dist-test-slave-hxxt krb5kdc[18353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:11Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:11Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:11.569855 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36037
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:36037
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:11.895309 18369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:11.895627 18369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:11.895777 18369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:11.904021 18369 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:11.904129 18369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:11.904193 18369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:11.904261 18369 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:11.904341 18369 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:11.915552 18369 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:36037
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36037
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:11.917207 18369 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:11.918596 18369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:11.926602 18374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:11.926573 18375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:11.927251 18377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:11.927845 18369 server_base.cc:1047] running on GCE node
I20251027 14:56:11.928694 18369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:11.930084 18369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:11.931298 18369 hybrid_clock.cc:648] HybridClock initialized: now 1761576971931230 us; error 39 us; skew 500 ppm
I20251027 14:56:11.931627 18369 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:11 dist-test-slave-hxxt krb5kdc[18353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:11.934594 18369 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:56:11.936189 18369 webserver.cc:492] Webserver started at http://127.0.104.62:38651/ using document root <none> and password file <none>
I20251027 14:56:11.936930 18369 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:11.937017 18369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:11.937362 18369 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:11.939880 18369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "e84861c8edef444c9909cd5b33ec35ec"
format_stamp: "Formatted at 2025-10-27 14:56:11 on dist-test-slave-hxxt"
server_key: "99aa2bafae67580d18a1eb4a43c7e00f"
server_key_iv: "d45aa3f91aa6dae2d1dfc581b95a521c"
server_key_version: "encryptionkey@0"
I20251027 14:56:11.940670 18369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "e84861c8edef444c9909cd5b33ec35ec"
format_stamp: "Formatted at 2025-10-27 14:56:11 on dist-test-slave-hxxt"
server_key: "99aa2bafae67580d18a1eb4a43c7e00f"
server_key_iv: "d45aa3f91aa6dae2d1dfc581b95a521c"
server_key_version: "encryptionkey@0"
I20251027 14:56:11.946250 18369 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:56:11.949672 18384 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:11.951045 18369 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20251027 14:56:11.951213 18369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "e84861c8edef444c9909cd5b33ec35ec"
format_stamp: "Formatted at 2025-10-27 14:56:11 on dist-test-slave-hxxt"
server_key: "99aa2bafae67580d18a1eb4a43c7e00f"
server_key_iv: "d45aa3f91aa6dae2d1dfc581b95a521c"
server_key_version: "encryptionkey@0"
I20251027 14:56:11.951423 18369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:11.963604 18369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:11.967272 18369 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:11.967516 18369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:11.986663 18369 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:36037
I20251027 14:56:11.986683 18435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:36037 every 8 connection(s)
I20251027 14:56:11.988102 18369 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:11.991748 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18369
I20251027 14:56:11.991979 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:11.992638 18436 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:11.992762 416 external_mini_cluster.cc:1468] Setting key b3800185844d7227328bc16069edca25
Oct 27 14:56:12 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:12.006418 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec: Bootstrap starting.
I20251027 14:56:12.010032 18436 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:12.011216 18436 log.cc:826] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:12.014813 18436 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec: No bootstrap required, opened a new log
I20251027 14:56:12.015216 18439 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:11.994739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50026 (local address 127.0.104.62:36037)
1027 14:56:11.995477 (+ 738us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:11.995487 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:11.995540 (+ 53us) server_negotiation.cc:408] Connection header received
1027 14:56:11.996832 (+ 1292us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:11.996857 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:11.997260 (+ 403us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:11.997643 (+ 383us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:11.999677 (+ 2034us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:12.001156 (+ 1479us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:12.002821 (+ 1665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:12.003379 (+ 558us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:12.007516 (+ 4137us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:12.007631 (+ 115us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:12.007649 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:12.007695 (+ 46us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:12.010765 (+ 3070us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:12.011731 (+ 966us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:12.011740 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:12.011748 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:12.011908 (+ 160us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:12.012292 (+ 384us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:12.012298 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:12.012301 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:12.012989 (+ 688us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:12.013222 (+ 233us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:12.013766 (+ 544us) server_negotiation.cc:300] Negotiation successful
1027 14:56:12.014166 (+ 400us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":339,"threads_started":1}
I20251027 14:56:12.020145 18436 raft_consensus.cc:359] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84861c8edef444c9909cd5b33ec35ec" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36037 } }
I20251027 14:56:12.020537 18436 raft_consensus.cc:385] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:12.020644 18436 raft_consensus.cc:740] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e84861c8edef444c9909cd5b33ec35ec, State: Initialized, Role: FOLLOWER
I20251027 14:56:12.021382 18436 consensus_queue.cc:260] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84861c8edef444c9909cd5b33ec35ec" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36037 } }
I20251027 14:56:12.021611 18436 raft_consensus.cc:399] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:12.021771 18436 raft_consensus.cc:493] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:12.021930 18436 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:12.024225 18436 raft_consensus.cc:515] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84861c8edef444c9909cd5b33ec35ec" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36037 } }
I20251027 14:56:12.024739 18436 leader_election.cc:304] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e84861c8edef444c9909cd5b33ec35ec; no voters:
I20251027 14:56:12.025342 18436 leader_election.cc:290] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:12.025472 18441 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:12.026124 18441 raft_consensus.cc:697] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [term 1 LEADER]: Becoming Leader. State: Replica: e84861c8edef444c9909cd5b33ec35ec, State: Running, Role: LEADER
I20251027 14:56:12.026574 18441 consensus_queue.cc:237] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84861c8edef444c9909cd5b33ec35ec" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36037 } }
I20251027 14:56:12.027284 18436 sys_catalog.cc:565] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:12.029884 18442 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e84861c8edef444c9909cd5b33ec35ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84861c8edef444c9909cd5b33ec35ec" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36037 } } }
I20251027 14:56:12.030170 18443 sys_catalog.cc:455] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [sys.catalog]: SysCatalogTable state changed. Reason: New leader e84861c8edef444c9909cd5b33ec35ec. Latest consensus state: current_term: 1 leader_uuid: "e84861c8edef444c9909cd5b33ec35ec" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e84861c8edef444c9909cd5b33ec35ec" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36037 } } }
I20251027 14:56:12.030587 18443 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:12.030397 18442 sys_catalog.cc:458] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:12.032335 18451 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:12.037626 18451 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:12.045909 18451 catalog_manager.cc:1357] Generated new cluster ID: 319ac963db6a4f969a98a60a8fc84a84
I20251027 14:56:12.046018 18451 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:12.063334 18451 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:12.065096 18451 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:12.095556 18451 catalog_manager.cc:6022] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec: Generated new TSK 0
I20251027 14:56:12.096751 18451 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:12.208817 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36037
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:12.542717 18464 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:12.542996 18464 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:12.543149 18464 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:12.551159 18464 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:12.551255 18464 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:12.551429 18464 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:56:12.561690 18464 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:36037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:12.563256 18464 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:12.564631 18464 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:12.573879 18469 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:12.573901 18470 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:12.574613 18472 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:12.575026 18464 server_base.cc:1047] running on GCE node
I20251027 14:56:12.575601 18464 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:12.576509 18464 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:12.577744 18464 hybrid_clock.cc:648] HybridClock initialized: now 1761576972577656 us; error 51 us; skew 500 ppm
I20251027 14:56:12.578044 18464 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:12 dist-test-slave-hxxt krb5kdc[18353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:12.581141 18464 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:56:12.582732 18464 webserver.cc:492] Webserver started at http://127.0.104.1:37239/ using document root <none> and password file <none>
I20251027 14:56:12.583431 18464 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:12.583525 18464 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:12.583896 18464 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:12.586508 18464 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "cc0cb655c7be47d3ad046b68bcda68c0"
format_stamp: "Formatted at 2025-10-27 14:56:12 on dist-test-slave-hxxt"
server_key: "0c8d20daefdb3ad39c6f4d13fdd98c29"
server_key_iv: "e6738dba1f172478bc419e01e0751aaa"
server_key_version: "encryptionkey@0"
I20251027 14:56:12.587255 18464 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "cc0cb655c7be47d3ad046b68bcda68c0"
format_stamp: "Formatted at 2025-10-27 14:56:12 on dist-test-slave-hxxt"
server_key: "0c8d20daefdb3ad39c6f4d13fdd98c29"
server_key_iv: "e6738dba1f172478bc419e01e0751aaa"
server_key_version: "encryptionkey@0"
I20251027 14:56:12.592497 18464 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:56:12.596052 18479 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:12.597506 18464 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:12.597680 18464 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "cc0cb655c7be47d3ad046b68bcda68c0"
format_stamp: "Formatted at 2025-10-27 14:56:12 on dist-test-slave-hxxt"
server_key: "0c8d20daefdb3ad39c6f4d13fdd98c29"
server_key_iv: "e6738dba1f172478bc419e01e0751aaa"
server_key_version: "encryptionkey@0"
I20251027 14:56:12.597949 18464 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:12.620430 18464 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:12.624655 18464 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:12.624928 18464 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:12.625962 18464 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:12.627560 18464 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:12.627656 18464 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:12.627763 18464 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:12.627835 18464 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:12.659137 18464 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45461
I20251027 14:56:12.659188 18591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45461 every 8 connection(s)
I20251027 14:56:12.660563 18464 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:56:12.665956 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18464
I20251027 14:56:12.666203 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:56:12.666921 416 external_mini_cluster.cc:1468] Setting key 26a70af0c5f110f9b6456739d7f3a603
Oct 27 14:56:12 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576972, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:12.680797 18594 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:12.663386 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:34791 (local address 127.0.104.62:36037)
1027 14:56:12.663817 (+ 431us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:12.663822 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:12.664746 (+ 924us) server_negotiation.cc:408] Connection header received
1027 14:56:12.666728 (+ 1982us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:12.666735 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:12.666809 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:12.666935 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:12.669009 (+ 2074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:12.669897 (+ 888us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:12.671379 (+ 1482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:12.671716 (+ 337us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:12.675060 (+ 3344us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:12.675093 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:12.675097 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:12.675130 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:12.677545 (+ 2415us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:12.678457 (+ 912us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:12.678464 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:12.678467 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:12.678566 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:12.678962 (+ 396us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:12.678967 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:12.678970 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:12.679258 (+ 288us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:12.679413 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:12.680168 (+ 755us) server_negotiation.cc:300] Negotiation successful
1027 14:56:12.680449 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":228,"threads_started":1}
I20251027 14:56:12.681685 18595 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:12.663833 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36037 (local address 127.0.104.1:34791)
1027 14:56:12.664585 (+ 752us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:12.664624 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:12.666373 (+ 1749us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:12.667209 (+ 836us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:12.667222 (+ 13us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:12.667984 (+ 762us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:12.668813 (+ 829us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:12.668838 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:12.670087 (+ 1249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:12.670094 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:12.671229 (+ 1135us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:12.671243 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:12.671501 (+ 258us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:12.672246 (+ 745us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:12.672276 (+ 30us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:12.674822 (+ 2546us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:12.677700 (+ 2878us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:12.677710 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:12.677729 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:12.678312 (+ 583us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:12.678697 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:12.678701 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:12.678704 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:12.678830 (+ 126us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:12.679378 (+ 548us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:12.679384 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:12.679737 (+ 353us) client_negotiation.cc:769] Sending connection context
1027 14:56:12.680034 (+ 297us) client_negotiation.cc:240] Negotiation successful
1027 14:56:12.680509 (+ 475us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":498,"thread_start_us":316,"threads_started":1}
I20251027 14:56:12.683511 18592 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36037
I20251027 14:56:12.683868 18592 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:12.684751 18592 heartbeater.cc:507] Master 127.0.104.62:36037 requested a full tablet report, sending...
I20251027 14:56:12.687311 18401 ts_manager.cc:194] Registered new tserver with Master: cc0cb655c7be47d3ad046b68bcda68c0 (127.0.104.1:45461)
I20251027 14:56:12.689545 18401 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:34791
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:12.763592 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36037
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:13.092304 18600 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:13.092657 18600 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:13.092789 18600 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:13.101006 18600 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:13.101115 18600 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:13.101300 18600 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:56:13.111801 18600 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:36037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:13.113420 18600 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:13.114846 18600 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:13.124667 18606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:13.124666 18605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:13.124948 18608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:13.125974 18600 server_base.cc:1047] running on GCE node
I20251027 14:56:13.126608 18600 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:13.127543 18600 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:13.128751 18600 hybrid_clock.cc:648] HybridClock initialized: now 1761576973128682 us; error 46 us; skew 500 ppm
I20251027 14:56:13.129034 18600 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:13 dist-test-slave-hxxt krb5kdc[18353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:13.132110 18600 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:56:13.133810 18600 webserver.cc:492] Webserver started at http://127.0.104.2:42103/ using document root <none> and password file <none>
I20251027 14:56:13.134531 18600 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:13.134618 18600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:13.134980 18600 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:13.137626 18600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "d659466fbe124428a0a0a163d8a13349"
format_stamp: "Formatted at 2025-10-27 14:56:13 on dist-test-slave-hxxt"
server_key: "90246f94314274ba7720ff0957654c0e"
server_key_iv: "9d9ca0d6457894c22ab9d12fa4da1fdf"
server_key_version: "encryptionkey@0"
I20251027 14:56:13.138393 18600 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "d659466fbe124428a0a0a163d8a13349"
format_stamp: "Formatted at 2025-10-27 14:56:13 on dist-test-slave-hxxt"
server_key: "90246f94314274ba7720ff0957654c0e"
server_key_iv: "9d9ca0d6457894c22ab9d12fa4da1fdf"
server_key_version: "encryptionkey@0"
I20251027 14:56:13.143536 18600 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251027 14:56:13.146916 18615 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:13.148305 18600 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:56:13.148499 18600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "d659466fbe124428a0a0a163d8a13349"
format_stamp: "Formatted at 2025-10-27 14:56:13 on dist-test-slave-hxxt"
server_key: "90246f94314274ba7720ff0957654c0e"
server_key_iv: "9d9ca0d6457894c22ab9d12fa4da1fdf"
server_key_version: "encryptionkey@0"
I20251027 14:56:13.148705 18600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:13.171905 18600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:13.176005 18600 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:13.176252 18600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:13.177287 18600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:13.178825 18600 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:13.178912 18600 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:13.178992 18600 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:13.179044 18600 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:13.211364 18600 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:45821
I20251027 14:56:13.211419 18727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:45821 every 8 connection(s)
I20251027 14:56:13.212857 18600 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:56:13.216506 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18600
I20251027 14:56:13.216686 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:56:13.217068 416 external_mini_cluster.cc:1468] Setting key ba0e45be1b685e905d0ad5237d4f6624
Oct 27 14:56:13 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:13.231941 18730 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.215664 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:53347 (local address 127.0.104.62:36037)
1027 14:56:13.216056 (+ 392us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:13.216061 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:13.217230 (+ 1169us) server_negotiation.cc:408] Connection header received
1027 14:56:13.218989 (+ 1759us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:13.218996 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:13.219078 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:13.219187 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:13.220929 (+ 1742us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.221816 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.222918 (+ 1102us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.223373 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.226292 (+ 2919us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:13.226336 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:13.226340 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:13.226371 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:13.229151 (+ 2780us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:13.229839 (+ 688us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:13.229845 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:13.229847 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:13.229925 (+ 78us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:13.230246 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:13.230250 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:13.230252 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:13.230491 (+ 239us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:13.230625 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:13.231355 (+ 730us) server_negotiation.cc:300] Negotiation successful
1027 14:56:13.231603 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":201,"threads_started":1}
I20251027 14:56:13.232910 18731 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.216128 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36037 (local address 127.0.104.2:53347)
1027 14:56:13.216970 (+ 842us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:13.217017 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:13.218663 (+ 1646us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:13.219445 (+ 782us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:13.219455 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:13.220020 (+ 565us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:13.220762 (+ 742us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.220784 (+ 22us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.221962 (+ 1178us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.221966 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:13.222779 (+ 813us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.222790 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.223064 (+ 274us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.223845 (+ 781us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:13.223869 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:13.226113 (+ 2244us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:13.229290 (+ 3177us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:13.229308 (+ 18us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:13.229326 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:13.229716 (+ 390us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:13.230022 (+ 306us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:13.230025 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:13.230027 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:13.230137 (+ 110us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:13.230608 (+ 471us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:13.230616 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:13.230939 (+ 323us) client_negotiation.cc:769] Sending connection context
1027 14:56:13.231247 (+ 308us) client_negotiation.cc:240] Negotiation successful
1027 14:56:13.231747 (+ 500us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":587,"thread_start_us":408,"threads_started":1}
I20251027 14:56:13.234231 18728 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36037
I20251027 14:56:13.234557 18728 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:13.235340 18728 heartbeater.cc:507] Master 127.0.104.62:36037 requested a full tablet report, sending...
I20251027 14:56:13.236943 18401 ts_manager.cc:194] Registered new tserver with Master: d659466fbe124428a0a0a163d8a13349 (127.0.104.2:45821)
I20251027 14:56:13.238049 18401 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:53347
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:13.311095 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36037
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37229
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:13.634945 18736 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:13.635260 18736 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:13.635396 18736 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:13.643512 18736 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:13.643637 18736 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:13.643821 18736 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:56:13.654378 18736 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37229
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:36037
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:13.655967 18736 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:13.657351 18736 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:13.666565 18741 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:13.666613 18742 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:13.667235 18744 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:13.667244 18736 server_base.cc:1047] running on GCE node
I20251027 14:56:13.667886 18736 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:13.668826 18736 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:13.670037 18736 hybrid_clock.cc:648] HybridClock initialized: now 1761576973669975 us; error 35 us; skew 500 ppm
I20251027 14:56:13.670318 18736 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:13 dist-test-slave-hxxt krb5kdc[18353](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:13.673461 18736 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:56:13.674933 18736 webserver.cc:492] Webserver started at http://127.0.104.3:36147/ using document root <none> and password file <none>
I20251027 14:56:13.675653 18736 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:13.675740 18736 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:13.676110 18736 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:13.678681 18736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "cacfa5d8a11440a589ad952ae20e028c"
format_stamp: "Formatted at 2025-10-27 14:56:13 on dist-test-slave-hxxt"
server_key: "6e546aa7e0f43b93d05a24434d1a648c"
server_key_iv: "1380c32a4848c12c4318b81ca6c9478c"
server_key_version: "encryptionkey@0"
I20251027 14:56:13.679445 18736 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "cacfa5d8a11440a589ad952ae20e028c"
format_stamp: "Formatted at 2025-10-27 14:56:13 on dist-test-slave-hxxt"
server_key: "6e546aa7e0f43b93d05a24434d1a648c"
server_key_iv: "1380c32a4848c12c4318b81ca6c9478c"
server_key_version: "encryptionkey@0"
I20251027 14:56:13.684809 18736 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20251027 14:56:13.688108 18751 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:13.689472 18736 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:13.689628 18736 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "cacfa5d8a11440a589ad952ae20e028c"
format_stamp: "Formatted at 2025-10-27 14:56:13 on dist-test-slave-hxxt"
server_key: "6e546aa7e0f43b93d05a24434d1a648c"
server_key_iv: "1380c32a4848c12c4318b81ca6c9478c"
server_key_version: "encryptionkey@0"
I20251027 14:56:13.689822 18736 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:13.692993 18592 heartbeater.cc:499] Master 127.0.104.62:36037 was elected leader, sending a full tablet report...
I20251027 14:56:13.705613 18736 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:13.713049 18736 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:13.713274 18736 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:13.714161 18736 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:13.715654 18736 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:13.715737 18736 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:13.715817 18736 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:13.715868 18736 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:13.747799 18736 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:36613
I20251027 14:56:13.747843 18863 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:36613 every 8 connection(s)
I20251027 14:56:13.749220 18736 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:56:13.753196 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18736
I20251027 14:56:13.753413 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:56:13.753939 416 external_mini_cluster.cc:1468] Setting key 447e408dcade11b9fa700e6967304ea6
Oct 27 14:56:13 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576973, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:13.768256 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.751882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:55339 (local address 127.0.104.62:36037)
1027 14:56:13.752349 (+ 467us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:13.752354 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:13.753448 (+ 1094us) server_negotiation.cc:408] Connection header received
1027 14:56:13.755373 (+ 1925us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:13.755379 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:13.755466 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:13.755577 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:13.757610 (+ 2033us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.758533 (+ 923us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.759597 (+ 1064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.760028 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.762839 (+ 2811us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:13.762875 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:13.762879 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:13.762909 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:13.765382 (+ 2473us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:13.766085 (+ 703us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:13.766091 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:13.766093 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:13.766164 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:13.766498 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:13.766502 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:13.766504 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:13.766754 (+ 250us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:13.766884 (+ 130us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:13.767637 (+ 753us) server_negotiation.cc:300] Negotiation successful
1027 14:56:13.767902 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":335,"thread_start_us":225,"threads_started":1}
I20251027 14:56:13.769217 18867 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.752438 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36037 (local address 127.0.104.3:55339)
1027 14:56:13.753266 (+ 828us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:13.753318 (+ 52us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:13.755024 (+ 1706us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:13.755840 (+ 816us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:13.755849 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:13.756649 (+ 800us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:13.757427 (+ 778us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.757446 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.758685 (+ 1239us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.758689 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:13.759465 (+ 776us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.759475 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.759730 (+ 255us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.760506 (+ 776us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:13.760532 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:13.762645 (+ 2113us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:13.765534 (+ 2889us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:13.765544 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:13.765564 (+ 20us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:13.765952 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:13.766262 (+ 310us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:13.766266 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:13.766268 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:13.766387 (+ 119us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:13.766878 (+ 491us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:13.766886 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:13.767204 (+ 318us) client_negotiation.cc:769] Sending connection context
1027 14:56:13.767515 (+ 311us) client_negotiation.cc:240] Negotiation successful
1027 14:56:13.768016 (+ 501us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":548,"thread_start_us":340,"threads_started":1}
I20251027 14:56:13.770538 18864 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36037
I20251027 14:56:13.770886 18864 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:13.771706 18864 heartbeater.cc:507] Master 127.0.104.62:36037 requested a full tablet report, sending...
I20251027 14:56:13.773206 18401 ts_manager.cc:194] Registered new tserver with Master: cacfa5d8a11440a589ad952ae20e028c (127.0.104.3:36613)
I20251027 14:56:13.774080 18401 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:55339
I20251027 14:56:13.781965 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:56:13.798545 18866 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.786704 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:50042 (local address 127.0.104.62:36037)
1027 14:56:13.786946 (+ 242us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:13.786953 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:13.787270 (+ 317us) server_negotiation.cc:408] Connection header received
1027 14:56:13.787414 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:13.787420 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:13.787499 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:13.787589 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:13.789217 (+ 1628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.790113 (+ 896us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.791405 (+ 1292us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.791895 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.793653 (+ 1758us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:13.793697 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:13.793701 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:13.793733 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:13.796264 (+ 2531us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:13.796890 (+ 626us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:13.796894 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:13.796897 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:13.796962 (+ 65us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:13.797294 (+ 332us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:13.797298 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:13.797300 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:13.797567 (+ 267us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:13.797766 (+ 199us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:13.798042 (+ 276us) server_negotiation.cc:300] Negotiation successful
1027 14:56:13.798228 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20251027 14:56:13.802266 18401 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50042:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:56:13.806928 18401 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:56:13.840329 18877 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.819970 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:45821 (local address 127.0.0.1:40864)
1027 14:56:13.820702 (+ 732us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:13.820745 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:13.820952 (+ 207us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:13.832316 (+ 11364us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:13.832324 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:13.832362 (+ 38us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:13.834185 (+ 1823us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.834198 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.836488 (+ 2290us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.836494 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:13.839382 (+ 2888us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.839399 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.839572 (+ 173us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.839648 (+ 76us) client_negotiation.cc:769] Sending connection context
1027 14:56:13.839798 (+ 150us) client_negotiation.cc:240] Negotiation successful
1027 14:56:13.839976 (+ 178us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":491,"thread_start_us":731,"threads_started":1}
I20251027 14:56:13.840991 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.820529 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60982 (local address 127.0.104.3:36613)
1027 14:56:13.830996 (+ 10467us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:13.831005 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:13.832229 (+ 1224us) server_negotiation.cc:408] Connection header received
1027 14:56:13.832349 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:13.832358 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:13.832700 (+ 342us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:13.832913 (+ 213us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:13.834367 (+ 1454us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.836251 (+ 1884us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.838519 (+ 2268us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.839648 (+ 1129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.839801 (+ 153us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:13.840542 (+ 741us) server_negotiation.cc:300] Negotiation successful
1027 14:56:13.840731 (+ 189us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10309,"thread_start_us":10169,"threads_started":1}
I20251027 14:56:13.840365 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.820472 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36613 (local address 127.0.0.1:60982)
1027 14:56:13.831973 (+ 11501us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:13.831992 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:13.832107 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:13.833234 (+ 1127us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:13.833239 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:13.833262 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:13.834184 (+ 922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.834198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.836541 (+ 2343us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.836546 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:13.838339 (+ 1793us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.838354 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.839925 (+ 1571us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.839948 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:56:13.840017 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:56:13.840114 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11398,"thread_start_us":11265,"threads_started":1}
I20251027 14:56:13.842619 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.820188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40864 (local address 127.0.104.2:45821)
1027 14:56:13.831259 (+ 11071us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:13.831269 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:13.831302 (+ 33us) server_negotiation.cc:408] Connection header received
1027 14:56:13.831423 (+ 121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:13.831432 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:13.831736 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:13.831980 (+ 244us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:13.834377 (+ 2397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.836212 (+ 1835us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.840788 (+ 4576us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.841886 (+ 1098us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.842041 (+ 155us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:13.842179 (+ 138us) server_negotiation.cc:300] Negotiation successful
1027 14:56:13.842367 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10950,"thread_start_us":10845,"threads_started":1}
I20251027 14:56:13.846200 18799 tablet_service.cc:1505] Processing CreateTablet for tablet f906376f7fb646e4b9eb53bc66a1f89f (DEFAULT_TABLE table=test-table [id=aab98688edcb4fbcaccbbf4b5add216a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:13.846973 18663 tablet_service.cc:1505] Processing CreateTablet for tablet f906376f7fb646e4b9eb53bc66a1f89f (DEFAULT_TABLE table=test-table [id=aab98688edcb4fbcaccbbf4b5add216a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:13.847839 18879 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.822045 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45461 (local address 127.0.0.1:49532)
1027 14:56:13.841454 (+ 19409us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:13.841473 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:13.841573 (+ 100us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:13.844028 (+ 2455us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:13.844034 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:13.844059 (+ 25us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:13.844492 (+ 433us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.844507 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.846046 (+ 1539us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.846050 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:13.847175 (+ 1125us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:13.847188 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.847330 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.847357 (+ 27us) client_negotiation.cc:769] Sending connection context
1027 14:56:13.847432 (+ 75us) client_negotiation.cc:240] Negotiation successful
1027 14:56:13.847563 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":19312,"thread_start_us":19719,"threads_started":1}
I20251027 14:56:13.847862 18799 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f906376f7fb646e4b9eb53bc66a1f89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:13.848434 18663 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f906376f7fb646e4b9eb53bc66a1f89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:13.849126 18882 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:13.822388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49532 (local address 127.0.104.1:45461)
1027 14:56:13.842355 (+ 19967us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:13.842364 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:13.842401 (+ 37us) server_negotiation.cc:408] Connection header received
1027 14:56:13.842511 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:13.842520 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:13.843537 (+ 1017us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:13.843743 (+ 206us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:13.844657 (+ 914us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.845892 (+ 1235us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:13.847323 (+ 1431us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:13.848595 (+ 1272us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:13.848708 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:13.848810 (+ 102us) server_negotiation.cc:300] Negotiation successful
1027 14:56:13.848938 (+ 128us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":19803,"thread_start_us":19708,"threads_started":1}
I20251027 14:56:13.852957 18527 tablet_service.cc:1505] Processing CreateTablet for tablet f906376f7fb646e4b9eb53bc66a1f89f (DEFAULT_TABLE table=test-table [id=aab98688edcb4fbcaccbbf4b5add216a]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:13.854252 18527 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f906376f7fb646e4b9eb53bc66a1f89f. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:13.862357 18883 tablet_bootstrap.cc:492] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Bootstrap starting.
I20251027 14:56:13.863247 18884 tablet_bootstrap.cc:492] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Bootstrap starting.
I20251027 14:56:13.865151 18885 tablet_bootstrap.cc:492] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Bootstrap starting.
I20251027 14:56:13.868228 18884 tablet_bootstrap.cc:654] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:13.868405 18885 tablet_bootstrap.cc:654] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:13.868880 18883 tablet_bootstrap.cc:654] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:13.869454 18885 log.cc:826] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:13.869580 18884 log.cc:826] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:13.870090 18883 log.cc:826] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:13.872915 18885 tablet_bootstrap.cc:492] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: No bootstrap required, opened a new log
I20251027 14:56:13.873128 18884 tablet_bootstrap.cc:492] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: No bootstrap required, opened a new log
I20251027 14:56:13.873201 18885 ts_tablet_manager.cc:1403] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20251027 14:56:13.873385 18883 tablet_bootstrap.cc:492] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: No bootstrap required, opened a new log
I20251027 14:56:13.873425 18884 ts_tablet_manager.cc:1403] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Time spent bootstrapping tablet: real 0.010s user 0.004s sys 0.003s
I20251027 14:56:13.873651 18883 ts_tablet_manager.cc:1403] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Time spent bootstrapping tablet: real 0.012s user 0.006s sys 0.001s
I20251027 14:56:13.878345 18884 raft_consensus.cc:359] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:13.878659 18884 raft_consensus.cc:385] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:13.878749 18884 raft_consensus.cc:740] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cacfa5d8a11440a589ad952ae20e028c, State: Initialized, Role: FOLLOWER
I20251027 14:56:13.879318 18884 consensus_queue.cc:260] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:13.880173 18885 raft_consensus.cc:359] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:13.880261 18883 raft_consensus.cc:359] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:13.880542 18864 heartbeater.cc:499] Master 127.0.104.62:36037 was elected leader, sending a full tablet report...
I20251027 14:56:13.880582 18885 raft_consensus.cc:385] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:13.880592 18883 raft_consensus.cc:385] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:13.880678 18885 raft_consensus.cc:740] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cc0cb655c7be47d3ad046b68bcda68c0, State: Initialized, Role: FOLLOWER
I20251027 14:56:13.880694 18883 raft_consensus.cc:740] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d659466fbe124428a0a0a163d8a13349, State: Initialized, Role: FOLLOWER
I20251027 14:56:13.881191 18884 ts_tablet_manager.cc:1434] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251027 14:56:13.881352 18885 consensus_queue.cc:260] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:13.881346 18883 consensus_queue.cc:260] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:13.882961 18883 ts_tablet_manager.cc:1434] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Time spent starting tablet: real 0.009s user 0.006s sys 0.003s
I20251027 14:56:13.883327 18885 ts_tablet_manager.cc:1434] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20251027 14:56:13.883498 18728 heartbeater.cc:499] Master 127.0.104.62:36037 was elected leader, sending a full tablet report...
W20251027 14:56:13.913007 18593 tablet.cc:2378] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:13.964956 18729 tablet.cc:2378] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:14.000780 18865 tablet.cc:2378] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:56:14.263185 18891 raft_consensus.cc:493] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:14.263437 18891 raft_consensus.cc:515] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:14.265208 18891 leader_election.cc:290] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d659466fbe124428a0a0a163d8a13349 (127.0.104.2:45821), cacfa5d8a11440a589ad952ae20e028c (127.0.104.3:36613)
I20251027 14:56:14.271919 18892 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.265460 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:36613 (local address 127.0.104.1:51791)
1027 14:56:14.266451 (+ 991us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:14.266475 (+ 24us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:14.266663 (+ 188us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:14.267051 (+ 388us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:14.267057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:14.267114 (+ 57us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:14.268058 (+ 944us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:14.268074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.269851 (+ 1777us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.269858 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:14.271121 (+ 1263us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:14.271136 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.271307 (+ 171us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.271340 (+ 33us) client_negotiation.cc:769] Sending connection context
1027 14:56:14.271411 (+ 71us) client_negotiation.cc:240] Negotiation successful
1027 14:56:14.271582 (+ 171us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":867,"spinlock_wait_cycles":32128,"thread_start_us":550,"threads_started":1}
I20251027 14:56:14.271941 18893 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.265661 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:45821 (local address 127.0.104.1:47327)
1027 14:56:14.266474 (+ 813us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:14.266488 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:14.266689 (+ 201us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:14.267051 (+ 362us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:14.267057 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:14.267089 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:14.268058 (+ 969us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:14.268074 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.269862 (+ 1788us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.269867 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:14.271146 (+ 1279us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:14.271160 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.271342 (+ 182us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.271367 (+ 25us) client_negotiation.cc:769] Sending connection context
1027 14:56:14.271434 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:56:14.271555 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":736,"spinlock_wait_cycles":29056,"thread_start_us":684,"threads_started":1}
I20251027 14:56:14.272758 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.265755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:51791 (local address 127.0.104.3:36613)
1027 14:56:14.265950 (+ 195us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:14.265957 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:14.266604 (+ 647us) server_negotiation.cc:408] Connection header received
1027 14:56:14.266826 (+ 222us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:14.266833 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:14.266913 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:14.267015 (+ 102us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:14.268229 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.269599 (+ 1370us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.271299 (+ 1700us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.272317 (+ 1018us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.272396 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:14.272473 (+ 77us) server_negotiation.cc:300] Negotiation successful
1027 14:56:14.272588 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:56:14.272861 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.266290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:47327 (local address 127.0.104.2:45821)
1027 14:56:14.266460 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:14.266467 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:14.266647 (+ 180us) server_negotiation.cc:408] Connection header received
1027 14:56:14.266834 (+ 187us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:14.266840 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:14.266912 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:14.267015 (+ 103us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:14.268229 (+ 1214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.269592 (+ 1363us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.271308 (+ 1716us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.272437 (+ 1129us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.272491 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:14.272564 (+ 73us) server_negotiation.cc:300] Negotiation successful
1027 14:56:14.272672 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:56:14.273643 18819 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f906376f7fb646e4b9eb53bc66a1f89f" candidate_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacfa5d8a11440a589ad952ae20e028c" is_pre_election: true
I20251027 14:56:14.273695 18683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f906376f7fb646e4b9eb53bc66a1f89f" candidate_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d659466fbe124428a0a0a163d8a13349" is_pre_election: true
I20251027 14:56:14.274092 18819 raft_consensus.cc:2468] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0cb655c7be47d3ad046b68bcda68c0 in term 0.
I20251027 14:56:14.274120 18683 raft_consensus.cc:2468] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate cc0cb655c7be47d3ad046b68bcda68c0 in term 0.
I20251027 14:56:14.274747 18482 leader_election.cc:304] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc0cb655c7be47d3ad046b68bcda68c0, d659466fbe124428a0a0a163d8a13349; no voters:
I20251027 14:56:14.275166 18891 raft_consensus.cc:2804] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:56:14.275326 18891 raft_consensus.cc:493] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:56:14.275411 18891 raft_consensus.cc:3060] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:14.277804 18891 raft_consensus.cc:515] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:14.278388 18891 leader_election.cc:290] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [CANDIDATE]: Term 1 election: Requested vote from peers d659466fbe124428a0a0a163d8a13349 (127.0.104.2:45821), cacfa5d8a11440a589ad952ae20e028c (127.0.104.3:36613)
I20251027 14:56:14.278817 18683 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f906376f7fb646e4b9eb53bc66a1f89f" candidate_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d659466fbe124428a0a0a163d8a13349"
I20251027 14:56:14.278883 18819 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "f906376f7fb646e4b9eb53bc66a1f89f" candidate_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "cacfa5d8a11440a589ad952ae20e028c"
I20251027 14:56:14.278980 18683 raft_consensus.cc:3060] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:14.279060 18819 raft_consensus.cc:3060] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:14.281338 18683 raft_consensus.cc:2468] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0cb655c7be47d3ad046b68bcda68c0 in term 1.
I20251027 14:56:14.281389 18819 raft_consensus.cc:2468] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate cc0cb655c7be47d3ad046b68bcda68c0 in term 1.
I20251027 14:56:14.281710 18482 leader_election.cc:304] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: cc0cb655c7be47d3ad046b68bcda68c0, d659466fbe124428a0a0a163d8a13349; no voters:
I20251027 14:56:14.282018 18891 raft_consensus.cc:2804] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:14.282420 18891 raft_consensus.cc:697] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 1 LEADER]: Becoming Leader. State: Replica: cc0cb655c7be47d3ad046b68bcda68c0, State: Running, Role: LEADER
I20251027 14:56:14.282932 18891 consensus_queue.cc:237] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } }
I20251027 14:56:14.288084 18399 catalog_manager.cc:5649] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 reported cstate change: term changed from 0 to 1, leader changed from <none> to cc0cb655c7be47d3ad046b68bcda68c0 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "cc0cb655c7be47d3ad046b68bcda68c0" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45461 } health_report { overall_health: HEALTHY } } }
I20251027 14:56:14.397872 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.392084 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49540 (local address 127.0.104.1:45461)
1027 14:56:14.392759 (+ 675us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:14.392765 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:14.392785 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:56:14.392958 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:14.392963 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:14.393067 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:14.393210 (+ 143us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:56:14.393912 (+ 702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.394827 (+ 915us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.396072 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.396511 (+ 439us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.396688 (+ 177us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:56:14.397262 (+ 574us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:56:14.397407 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:14.397511 (+ 104us) server_negotiation.cc:300] Negotiation successful
1027 14:56:14.397613 (+ 102us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":466,"threads_started":1}
I20251027 14:56:14.408563 18683 raft_consensus.cc:1275] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 1 FOLLOWER]: Refusing update from remote peer cc0cb655c7be47d3ad046b68bcda68c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:14.408854 18819 raft_consensus.cc:1275] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 1 FOLLOWER]: Refusing update from remote peer cc0cb655c7be47d3ad046b68bcda68c0: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:14.409690 18894 consensus_queue.cc:1048] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d659466fbe124428a0a0a163d8a13349" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 45821 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:14.409958 18891 consensus_queue.cc:1048] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [LEADER]: Connected to new peer: Peer: permanent_uuid: "cacfa5d8a11440a589ad952ae20e028c" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 36613 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:14.418762 18899 mvcc.cc:204] Tried to move back new op lower bound from 7215419287164313600 to 7215419286666461184. Current Snapshot: MvccSnapshot[applied={T|T < 7215419287164313600}]
I20251027 14:56:14.438043 18399 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50042:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:14.438246 18399 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:50042:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:14.441543 18399 catalog_manager.cc:5936] T 00000000000000000000000000000000 P e84861c8edef444c9909cd5b33ec35ec: Sending DeleteTablet for 3 replicas of tablet f906376f7fb646e4b9eb53bc66a1f89f
I20251027 14:56:14.442716 18663 tablet_service.cc:1552] Processing DeleteTablet for tablet f906376f7fb646e4b9eb53bc66a1f89f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:14 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:40864
I20251027 14:56:14.442734 18799 tablet_service.cc:1552] Processing DeleteTablet for tablet f906376f7fb646e4b9eb53bc66a1f89f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:14 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:60982
I20251027 14:56:14.442740 18527 tablet_service.cc:1552] Processing DeleteTablet for tablet f906376f7fb646e4b9eb53bc66a1f89f with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:14 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:49532
I20251027 14:56:14.448450 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18464
I20251027 14:56:14.455221 18911 tablet_replica.cc:333] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: stopping tablet replica
I20251027 14:56:14.455880 18911 raft_consensus.cc:2243] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:14.456548 18911 raft_consensus.cc:2272] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:14.458544 18911 ts_tablet_manager.cc:1916] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:14.466317 18911 ts_tablet_manager.cc:1929] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:14.466598 18911 log.cc:1199] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/f906376f7fb646e4b9eb53bc66a1f89f
I20251027 14:56:14.467192 18911 ts_tablet_manager.cc:1950] T f906376f7fb646e4b9eb53bc66a1f89f P d659466fbe124428a0a0a163d8a13349: Deleting consensus metadata
I20251027 14:56:14.469329 18387 catalog_manager.cc:4985] TS d659466fbe124428a0a0a163d8a13349 (127.0.104.2:45821): tablet f906376f7fb646e4b9eb53bc66a1f89f (table test-table [id=aab98688edcb4fbcaccbbf4b5add216a]) successfully deleted
I20251027 14:56:14.471683 18912 tablet_replica.cc:333] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: stopping tablet replica
I20251027 14:56:14.472288 18912 raft_consensus.cc:2243] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:56:14.473160 18912 raft_consensus.cc:2272] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:14.475930 18913 tablet_replica.cc:333] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: stopping tablet replica
I20251027 14:56:14.476307 18912 ts_tablet_manager.cc:1916] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:14.476368 18913 raft_consensus.cc:2243] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:14.476886 18913 raft_consensus.cc:2272] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c [term 1 FOLLOWER]: Raft consensus is shut down!
Oct 27 14:56:14 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:56:14.478657 18913 ts_tablet_manager.cc:1916] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:14.484055 18912 ts_tablet_manager.cc:1929] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:14.484221 18912 log.cc:1199] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/f906376f7fb646e4b9eb53bc66a1f89f
I20251027 14:56:14.484737 18912 ts_tablet_manager.cc:1950] T f906376f7fb646e4b9eb53bc66a1f89f P cc0cb655c7be47d3ad046b68bcda68c0: Deleting consensus metadata
I20251027 14:56:14.486287 18913 ts_tablet_manager.cc:1929] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:14.486441 18913 log.cc:1199] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_0.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/f906376f7fb646e4b9eb53bc66a1f89f
I20251027 14:56:14.486582 18385 catalog_manager.cc:4985] TS cc0cb655c7be47d3ad046b68bcda68c0 (127.0.104.1:45461): tablet f906376f7fb646e4b9eb53bc66a1f89f (table test-table [id=aab98688edcb4fbcaccbbf4b5add216a]) successfully deleted
I20251027 14:56:14.486902 18913 ts_tablet_manager.cc:1950] T f906376f7fb646e4b9eb53bc66a1f89f P cacfa5d8a11440a589ad952ae20e028c: Deleting consensus metadata
I20251027 14:56:14.488175 18898 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.449904 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49556 (local address 127.0.104.1:45461)
1027 14:56:14.450046 (+ 142us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:14.450052 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:14.467171 (+ 17119us) server_negotiation.cc:408] Connection header received
1027 14:56:14.467732 (+ 561us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:14.467741 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:14.467859 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:14.467942 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:14.470306 (+ 2364us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.471876 (+ 1570us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.473929 (+ 2053us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.474440 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.479973 (+ 5533us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:14.480025 (+ 52us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:14.480043 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:14.480088 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:14.484603 (+ 4515us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:14.485530 (+ 927us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:14.485540 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:14.485544 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:14.485638 (+ 94us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:14.486052 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:14.486057 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:14.486061 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:14.486561 (+ 500us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:14.486902 (+ 341us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:14.487549 (+ 647us) server_negotiation.cc:300] Negotiation successful
1027 14:56:14.487816 (+ 267us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":32}
I20251027 14:56:14.488925 18587 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:14.488981 18388 catalog_manager.cc:4985] TS cacfa5d8a11440a589ad952ae20e028c (127.0.104.3:36613): tablet f906376f7fb646e4b9eb53bc66a1f89f (table test-table [id=aab98688edcb4fbcaccbbf4b5add216a]) successfully deleted
I20251027 14:56:14.586710 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18464
I20251027 14:56:14.601811 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18600
Oct 27 14:56:14 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:56:14.616057 18878 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.602581 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40866 (local address 127.0.104.2:45821)
1027 14:56:14.602778 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:14.602783 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:14.602803 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:56:14.602963 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:14.602968 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:14.603043 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:14.603135 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:14.604551 (+ 1416us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.605451 (+ 900us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.606652 (+ 1201us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.607095 (+ 443us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.610607 (+ 3512us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:14.610651 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:14.610668 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:14.610710 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:14.613479 (+ 2769us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:14.614141 (+ 662us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:14.614148 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:14.614151 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:14.614240 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:14.614560 (+ 320us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:14.614565 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:14.614568 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:14.614893 (+ 325us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:14.615192 (+ 299us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:14.615483 (+ 291us) server_negotiation.cc:300] Negotiation successful
1027 14:56:14.615732 (+ 249us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:56:14.617019 18723 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:14.705821 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18600
I20251027 14:56:14.719774 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18736
Oct 27 14:56:14 dist-test-slave-hxxt krb5kdc[18353](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576971, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:56:14.734237 18880 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:14.720605 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60986 (local address 127.0.104.3:36613)
1027 14:56:14.720811 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:14.720816 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:14.720835 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:56:14.720893 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:14.720898 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:14.721001 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:14.721086 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:14.722549 (+ 1463us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.723423 (+ 874us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:14.724622 (+ 1199us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:14.724988 (+ 366us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:14.728517 (+ 3529us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:14.728559 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:14.728575 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:14.728617 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:14.731478 (+ 2861us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:14.732143 (+ 665us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:14.732151 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:14.732154 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:14.732241 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:14.732745 (+ 504us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:14.732749 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:14.732751 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:14.733069 (+ 318us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:14.733324 (+ 255us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:14.733708 (+ 384us) server_negotiation.cc:300] Negotiation successful
1027 14:56:14.733939 (+ 231us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
I20251027 14:56:14.734977 18859 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:14.823410 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18736
I20251027 14:56:14.836889 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18369
I20251027 14:56:14.838106 18431 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:14.947432 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18369
2025-10-27T14:56:14Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/0 (5542 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/1
2025-10-27T14:56:14Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:14Z Disabled control of system clock
I20251027 14:56:15.018224 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35153
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:35153
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251027 14:56:15.344867 18922 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:15.345180 18922 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:15.345264 18922 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:15.353402 18922 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:15.353510 18922 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:15.353561 18922 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:15.353610 18922 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:15.353667 18922 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:15.363534 18922 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:35153
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:35153
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:15.365156 18922 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:15.366928 18922 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:15.375162 18928 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:15.375473 18930 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:15.375124 18927 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:15.376098 18922 server_base.cc:1047] running on GCE node
I20251027 14:56:15.376852 18922 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:15.378160 18922 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:15.379366 18922 hybrid_clock.cc:648] HybridClock initialized: now 1761576975379301 us; error 39 us; skew 500 ppm
I20251027 14:56:15.379700 18922 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:15.381852 18922 webserver.cc:492] Webserver started at http://127.0.104.62:45107/ using document root <none> and password file <none>
I20251027 14:56:15.382567 18922 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:15.382669 18922 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:15.383030 18922 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:15.385609 18922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "70aa9b77c3bb4fff94b58f9c0823f64d"
format_stamp: "Formatted at 2025-10-27 14:56:15 on dist-test-slave-hxxt"
server_key: "ba1c45015e78ed7ff8df94ee8312ede9"
server_key_iv: "800f9e63ceb6cdea1591a8a1319a3632"
server_key_version: "encryptionkey@0"
I20251027 14:56:15.386377 18922 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "70aa9b77c3bb4fff94b58f9c0823f64d"
format_stamp: "Formatted at 2025-10-27 14:56:15 on dist-test-slave-hxxt"
server_key: "ba1c45015e78ed7ff8df94ee8312ede9"
server_key_iv: "800f9e63ceb6cdea1591a8a1319a3632"
server_key_version: "encryptionkey@0"
I20251027 14:56:15.391742 18922 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:56:15.395170 18936 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:15.396502 18922 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:15.396661 18922 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "70aa9b77c3bb4fff94b58f9c0823f64d"
format_stamp: "Formatted at 2025-10-27 14:56:15 on dist-test-slave-hxxt"
server_key: "ba1c45015e78ed7ff8df94ee8312ede9"
server_key_iv: "800f9e63ceb6cdea1591a8a1319a3632"
server_key_version: "encryptionkey@0"
I20251027 14:56:15.396853 18922 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:15.422642 18922 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:15.423539 18922 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:15.423803 18922 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:15.442305 18922 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:35153
I20251027 14:56:15.442329 18987 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:35153 every 8 connection(s)
I20251027 14:56:15.443755 18922 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:15.447834 18988 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:15.449949 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 18922
I20251027 14:56:15.450139 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:15.450505 416 external_mini_cluster.cc:1468] Setting key 90366f2b7452c755d2f5bec4a938c7c3
I20251027 14:56:15.460783 18988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d: Bootstrap starting.
I20251027 14:56:15.463860 18991 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:15.452351 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42236 (local address 127.0.104.62:35153)
1027 14:56:15.453447 (+ 1096us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:15.453461 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:15.453515 (+ 54us) server_negotiation.cc:408] Connection header received
1027 14:56:15.455054 (+ 1539us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:15.455084 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:15.455696 (+ 612us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:15.456196 (+ 500us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:15.456904 (+ 708us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:15.458934 (+ 2030us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:15.460141 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:15.460774 (+ 633us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:15.461025 (+ 251us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:15.461127 (+ 102us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:15.461146 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:15.461828 (+ 682us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:15.461889 (+ 61us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:15.461914 (+ 25us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:15.462036 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:15.462218 (+ 182us) server_negotiation.cc:300] Negotiation successful
1027 14:56:15.462561 (+ 343us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":557,"thread_start_us":348,"threads_started":1}
I20251027 14:56:15.464754 18988 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:15.465826 18988 log.cc:826] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:15.468556 18988 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d: No bootstrap required, opened a new log
I20251027 14:56:15.473713 18988 raft_consensus.cc:359] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35153 } }
I20251027 14:56:15.474050 18988 raft_consensus.cc:385] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:15.474160 18988 raft_consensus.cc:740] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 70aa9b77c3bb4fff94b58f9c0823f64d, State: Initialized, Role: FOLLOWER
I20251027 14:56:15.474798 18988 consensus_queue.cc:260] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35153 } }
I20251027 14:56:15.475066 18988 raft_consensus.cc:399] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:15.475172 18988 raft_consensus.cc:493] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:15.475291 18988 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:15.477515 18988 raft_consensus.cc:515] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35153 } }
I20251027 14:56:15.477964 18988 leader_election.cc:304] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 70aa9b77c3bb4fff94b58f9c0823f64d; no voters:
I20251027 14:56:15.478477 18988 leader_election.cc:290] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:15.478588 18993 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:15.479219 18993 raft_consensus.cc:697] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [term 1 LEADER]: Becoming Leader. State: Replica: 70aa9b77c3bb4fff94b58f9c0823f64d, State: Running, Role: LEADER
I20251027 14:56:15.479673 18993 consensus_queue.cc:237] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35153 } }
I20251027 14:56:15.480332 18988 sys_catalog.cc:565] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:15.482345 18994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35153 } } }
I20251027 14:56:15.482671 18994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:15.484464 18995 sys_catalog.cc:455] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 70aa9b77c3bb4fff94b58f9c0823f64d. Latest consensus state: current_term: 1 leader_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "70aa9b77c3bb4fff94b58f9c0823f64d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 35153 } } }
I20251027 14:56:15.484760 18995 sys_catalog.cc:458] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:15.485329 19000 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:15.490514 19000 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:15.498689 19000 catalog_manager.cc:1357] Generated new cluster ID: 34c27a4941f64a99a68a5187c1b44598
I20251027 14:56:15.498811 19000 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:15.520432 19000 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:15.521677 19000 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:15.532986 19000 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d: Generated new TSK 0
I20251027 14:56:15.533878 19000 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:56:15.547883 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35153
--builtin_ntp_servers=127.0.104.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251027 14:56:15.876849 19012 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:15.877140 19012 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:15.877262 19012 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:15.885447 19012 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:15.885535 19012 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:15.885695 19012 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:56:15.895431 19012 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:15.897136 19012 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:15.898953 19012 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:15.908191 19018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:15.908265 19017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:15.908998 19020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:15.909361 19012 server_base.cc:1047] running on GCE node
I20251027 14:56:15.909960 19012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:15.910884 19012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:15.912117 19012 hybrid_clock.cc:648] HybridClock initialized: now 1761576975912039 us; error 52 us; skew 500 ppm
I20251027 14:56:15.912525 19012 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:15.914768 19012 webserver.cc:492] Webserver started at http://127.0.104.1:46385/ using document root <none> and password file <none>
I20251027 14:56:15.915571 19012 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:15.915695 19012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:15.916100 19012 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:15.918787 19012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "855bbee394c6416e9a2b49c5238faf2f"
format_stamp: "Formatted at 2025-10-27 14:56:15 on dist-test-slave-hxxt"
server_key: "c7843343edd134058cc162bdc67391ca"
server_key_iv: "c7809475215da89d9f4134ad4144b39a"
server_key_version: "encryptionkey@0"
I20251027 14:56:15.919576 19012 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "855bbee394c6416e9a2b49c5238faf2f"
format_stamp: "Formatted at 2025-10-27 14:56:15 on dist-test-slave-hxxt"
server_key: "c7843343edd134058cc162bdc67391ca"
server_key_iv: "c7809475215da89d9f4134ad4144b39a"
server_key_version: "encryptionkey@0"
I20251027 14:56:15.924913 19012 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:56:15.928274 19026 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:15.929690 19012 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:56:15.929847 19012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "855bbee394c6416e9a2b49c5238faf2f"
format_stamp: "Formatted at 2025-10-27 14:56:15 on dist-test-slave-hxxt"
server_key: "c7843343edd134058cc162bdc67391ca"
server_key_iv: "c7809475215da89d9f4134ad4144b39a"
server_key_version: "encryptionkey@0"
I20251027 14:56:15.930033 19012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:15.951836 19012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:15.952725 19012 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:15.952997 19012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:15.953936 19012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:15.955503 19012 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:15.955586 19012 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:15.955708 19012 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:15.955758 19012 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:15.986436 19012 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:45503
I20251027 14:56:15.986506 19138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:45503 every 8 connection(s)
I20251027 14:56:15.987782 19012 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:56:15.990578 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19012
I20251027 14:56:15.990780 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:56:15.991190 416 external_mini_cluster.cc:1468] Setting key edae1969c7fb1e2fa6eb4897ec59bbe0
I20251027 14:56:15.993888 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35153
--builtin_ntp_servers=127.0.104.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251027 14:56:15.999374 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:15.990446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:44335 (local address 127.0.104.62:35153)
1027 14:56:15.990871 (+ 425us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:15.990877 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:15.992261 (+ 1384us) server_negotiation.cc:408] Connection header received
1027 14:56:15.993797 (+ 1536us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:15.993804 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:15.993885 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:15.994009 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:15.995294 (+ 1285us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:15.996521 (+ 1227us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:15.997593 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:15.997982 (+ 389us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:15.998581 (+ 599us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:15.998623 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:15.998628 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:15.998648 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:15.998689 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:15.998704 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:15.998788 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:15.998991 (+ 203us) server_negotiation.cc:300] Negotiation successful
1027 14:56:15.999096 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":311,"thread_start_us":209,"threads_started":1}
I20251027 14:56:16.000229 19142 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:15.991038 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35153 (local address 127.0.104.1:44335)
1027 14:56:15.992075 (+ 1037us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:15.992116 (+ 41us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:15.993519 (+ 1403us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:15.994282 (+ 763us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:15.994291 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:15.994384 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:15.995114 (+ 730us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:15.995133 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:15.996677 (+ 1544us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:15.996682 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:15.997472 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:15.997482 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:15.997680 (+ 198us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:15.998333 (+ 653us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:15.998335 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:15.998364 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:15.998427 (+ 63us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:15.998789 (+ 362us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:15.998794 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:15.998799 (+ 5us) client_negotiation.cc:769] Sending connection context
1027 14:56:15.998968 (+ 169us) client_negotiation.cc:240] Negotiation successful
1027 14:56:15.999260 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":529,"thread_start_us":335,"threads_started":1}
I20251027 14:56:16.001943 19139 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35153
I20251027 14:56:16.002266 19139 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:16.003042 19139 heartbeater.cc:507] Master 127.0.104.62:35153 requested a full tablet report, sending...
I20251027 14:56:16.005337 18953 ts_manager.cc:194] Registered new tserver with Master: 855bbee394c6416e9a2b49c5238faf2f (127.0.104.1:45503)
I20251027 14:56:16.007297 18953 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:44335
W20251027 14:56:16.315572 19143 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:16.315856 19143 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:16.315943 19143 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:16.323865 19143 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:16.323967 19143 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:16.324092 19143 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:56:16.333544 19143 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:16.335158 19143 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:16.336966 19143 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:16.346144 19148 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:16.346177 19149 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:16.346877 19151 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:16.347262 19143 server_base.cc:1047] running on GCE node
I20251027 14:56:16.347852 19143 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:16.348760 19143 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:16.349978 19143 hybrid_clock.cc:648] HybridClock initialized: now 1761576976349911 us; error 41 us; skew 500 ppm
I20251027 14:56:16.350260 19143 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:16.352471 19143 webserver.cc:492] Webserver started at http://127.0.104.2:41365/ using document root <none> and password file <none>
I20251027 14:56:16.353181 19143 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:16.353296 19143 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:16.353645 19143 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:16.356178 19143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "22fd428d276b4cfb84277345f743baff"
format_stamp: "Formatted at 2025-10-27 14:56:16 on dist-test-slave-hxxt"
server_key: "3295416faf7e3ed786c1e2e9f69d1cdd"
server_key_iv: "4e500d2575b2ccadd4deae46201f103f"
server_key_version: "encryptionkey@0"
I20251027 14:56:16.356981 19143 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "22fd428d276b4cfb84277345f743baff"
format_stamp: "Formatted at 2025-10-27 14:56:16 on dist-test-slave-hxxt"
server_key: "3295416faf7e3ed786c1e2e9f69d1cdd"
server_key_iv: "4e500d2575b2ccadd4deae46201f103f"
server_key_version: "encryptionkey@0"
I20251027 14:56:16.362183 19143 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:56:16.365711 19157 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:16.367169 19143 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:56:16.367342 19143 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "22fd428d276b4cfb84277345f743baff"
format_stamp: "Formatted at 2025-10-27 14:56:16 on dist-test-slave-hxxt"
server_key: "3295416faf7e3ed786c1e2e9f69d1cdd"
server_key_iv: "4e500d2575b2ccadd4deae46201f103f"
server_key_version: "encryptionkey@0"
I20251027 14:56:16.367535 19143 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:16.389178 19143 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:16.390074 19143 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:16.390334 19143 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:16.391280 19143 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:16.392941 19143 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:16.393029 19143 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:16.393146 19143 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:16.393196 19143 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:16.425221 19143 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35709
I20251027 14:56:16.425277 19269 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35709 every 8 connection(s)
I20251027 14:56:16.426609 19143 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:56:16.435040 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19143
I20251027 14:56:16.435217 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:56:16.435606 416 external_mini_cluster.cc:1468] Setting key 18bf6b45855414fdacebc8c3dcb736f7
I20251027 14:56:16.437880 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.429175 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:50987 (local address 127.0.104.62:35153)
1027 14:56:16.429374 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:16.429381 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:16.430859 (+ 1478us) server_negotiation.cc:408] Connection header received
1027 14:56:16.432317 (+ 1458us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:16.432322 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:16.432424 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:16.432508 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:16.433741 (+ 1233us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.434628 (+ 887us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.435812 (+ 1184us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.436243 (+ 431us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.436998 (+ 755us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:16.437035 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:16.437040 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:16.437057 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:16.437094 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:16.437109 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:16.437198 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:16.437471 (+ 273us) server_negotiation.cc:300] Negotiation successful
1027 14:56:16.437592 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:56:16.438506 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35153
--builtin_ntp_servers=127.0.104.20:43593
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251027 14:56:16.438858 19272 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.429766 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35153 (local address 127.0.104.2:50987)
1027 14:56:16.430707 (+ 941us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:16.430744 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:16.432070 (+ 1326us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:16.432735 (+ 665us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:16.432751 (+ 16us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:16.432842 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:16.433569 (+ 727us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.433586 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.434777 (+ 1191us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.434782 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:16.435629 (+ 847us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.435641 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.435889 (+ 248us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.436739 (+ 850us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:16.436742 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:16.436773 (+ 31us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:16.436835 (+ 62us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:16.437200 (+ 365us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:16.437208 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:16.437215 (+ 7us) client_negotiation.cc:769] Sending connection context
1027 14:56:16.437452 (+ 237us) client_negotiation.cc:240] Negotiation successful
1027 14:56:16.437839 (+ 387us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":499,"thread_start_us":319,"threads_started":1}
I20251027 14:56:16.440270 19270 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35153
I20251027 14:56:16.440673 19270 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:16.441473 19270 heartbeater.cc:507] Master 127.0.104.62:35153 requested a full tablet report, sending...
I20251027 14:56:16.443163 18953 ts_manager.cc:194] Registered new tserver with Master: 22fd428d276b4cfb84277345f743baff (127.0.104.2:35709)
I20251027 14:56:16.444133 18953 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:50987
W20251027 14:56:16.760694 19273 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:16.760974 19273 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:16.761098 19273 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:16.769129 19273 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:16.769217 19273 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:16.769373 19273 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:56:16.779007 19273 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43593
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:35153
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:16.780639 19273 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:16.782449 19273 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:16.791615 19278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:16.791644 19279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:16.792596 19281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:16.792833 19273 server_base.cc:1047] running on GCE node
I20251027 14:56:16.793440 19273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:16.794375 19273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:16.795620 19273 hybrid_clock.cc:648] HybridClock initialized: now 1761576976795497 us; error 101 us; skew 500 ppm
I20251027 14:56:16.795921 19273 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:16.798166 19273 webserver.cc:492] Webserver started at http://127.0.104.3:41747/ using document root <none> and password file <none>
I20251027 14:56:16.798938 19273 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:16.799055 19273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:16.799386 19273 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:16.802160 19273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "13429566430b46fa95bd3a625cbeab39"
format_stamp: "Formatted at 2025-10-27 14:56:16 on dist-test-slave-hxxt"
server_key: "a57fd3ff01ba5e7062760022dad4d67f"
server_key_iv: "d4c80b27b514fe619c133eb90c92c0a7"
server_key_version: "encryptionkey@0"
I20251027 14:56:16.802898 19273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "13429566430b46fa95bd3a625cbeab39"
format_stamp: "Formatted at 2025-10-27 14:56:16 on dist-test-slave-hxxt"
server_key: "a57fd3ff01ba5e7062760022dad4d67f"
server_key_iv: "d4c80b27b514fe619c133eb90c92c0a7"
server_key_version: "encryptionkey@0"
I20251027 14:56:16.808152 19273 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:56:16.811705 19287 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:16.813205 19273 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:16.813370 19273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "13429566430b46fa95bd3a625cbeab39"
format_stamp: "Formatted at 2025-10-27 14:56:16 on dist-test-slave-hxxt"
server_key: "a57fd3ff01ba5e7062760022dad4d67f"
server_key_iv: "d4c80b27b514fe619c133eb90c92c0a7"
server_key_version: "encryptionkey@0"
I20251027 14:56:16.813570 19273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:16.833092 19273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:16.834197 19273 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:16.834494 19273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:16.835521 19273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:16.837146 19273 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:16.837225 19273 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:16.837308 19273 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:16.837352 19273 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:16.868451 19273 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:46815
I20251027 14:56:16.868460 19399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:46815 every 8 connection(s)
I20251027 14:56:16.869836 19273 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:56:16.872450 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19273
I20251027 14:56:16.872653 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:56:16.873123 416 external_mini_cluster.cc:1468] Setting key 8f55f9d52b90745a485c2a08f0fefc55
I20251027 14:56:16.881183 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.872594 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:44365 (local address 127.0.104.62:35153)
1027 14:56:16.872823 (+ 229us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:16.872829 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:16.874432 (+ 1603us) server_negotiation.cc:408] Connection header received
1027 14:56:16.875947 (+ 1515us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:16.875951 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:16.876018 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:16.876098 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:16.877367 (+ 1269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.878235 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.879359 (+ 1124us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.879688 (+ 329us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.880356 (+ 668us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:16.880409 (+ 53us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:16.880415 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:16.880434 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:16.880466 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:16.880479 (+ 13us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:16.880554 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:16.880805 (+ 251us) server_negotiation.cc:300] Negotiation successful
1027 14:56:16.880909 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:56:16.882144 19402 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.873185 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:35153 (local address 127.0.104.3:44365)
1027 14:56:16.874275 (+ 1090us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:16.874317 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:16.875689 (+ 1372us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:16.876308 (+ 619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:16.876317 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:16.876452 (+ 135us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:16.877207 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.877226 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.878385 (+ 1159us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.878389 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:16.879224 (+ 835us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.879236 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.879439 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.880106 (+ 667us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:16.880109 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:16.880137 (+ 28us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:16.880202 (+ 65us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:16.880574 (+ 372us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:16.880582 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:16.880587 (+ 5us) client_negotiation.cc:769] Sending connection context
1027 14:56:16.880803 (+ 216us) client_negotiation.cc:240] Negotiation successful
1027 14:56:16.881142 (+ 339us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":576,"thread_start_us":365,"threads_started":1}
I20251027 14:56:16.883548 19400 heartbeater.cc:344] Connected to a master server at 127.0.104.62:35153
I20251027 14:56:16.883898 19400 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:16.884737 19400 heartbeater.cc:507] Master 127.0.104.62:35153 requested a full tablet report, sending...
I20251027 14:56:16.886274 18953 ts_manager.cc:194] Registered new tserver with Master: 13429566430b46fa95bd3a625cbeab39 (127.0.104.3:46815)
I20251027 14:56:16.887298 18953 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:44365
I20251027 14:56:16.888355 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:56:16.897719 19141 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.892511 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42252 (local address 127.0.104.62:35153)
1027 14:56:16.892678 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:16.892683 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:16.892998 (+ 315us) server_negotiation.cc:408] Connection header received
1027 14:56:16.893152 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:16.893156 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:16.893217 (+ 61us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:16.893296 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:16.894090 (+ 794us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.894959 (+ 869us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.896208 (+ 1249us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.896630 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.897078 (+ 448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:16.897105 (+ 27us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:16.897109 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:16.897124 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:16.897156 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:16.897167 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:16.897239 (+ 72us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:16.897406 (+ 167us) server_negotiation.cc:300] Negotiation successful
1027 14:56:16.897498 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:56:16.901362 18953 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:42252:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:56:16.905956 18953 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:56:16.937227 19412 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.918198 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:59916)
1027 14:56:16.919556 (+ 1358us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:16.919598 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:16.919787 (+ 189us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:16.931272 (+ 11485us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:16.931282 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:16.931326 (+ 44us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:16.932324 (+ 998us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.932341 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.934406 (+ 2065us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.934409 (+ 3us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:16.936191 (+ 1782us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.936210 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.936415 (+ 205us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.936491 (+ 76us) client_negotiation.cc:769] Sending connection context
1027 14:56:16.936664 (+ 173us) client_negotiation.cc:240] Negotiation successful
1027 14:56:16.936870 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1075,"thread_start_us":1593,"threads_started":1}
I20251027 14:56:16.937297 19415 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.918591 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:35709 (local address 127.0.0.1:60268)
1027 14:56:16.929910 (+ 11319us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:16.929933 (+ 23us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:16.930069 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:16.931956 (+ 1887us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:16.931962 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:16.931994 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:16.932723 (+ 729us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.932736 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.934839 (+ 2103us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.934845 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:16.936650 (+ 1805us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.936663 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.936833 (+ 170us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.936859 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:56:16.936921 (+ 62us) client_negotiation.cc:240] Negotiation successful
1027 14:56:16.937024 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":11196,"thread_start_us":13042,"threads_started":1}
I20251027 14:56:16.938311 19414 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.918927 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:46815 (local address 127.0.0.1:55824)
1027 14:56:16.931995 (+ 13068us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:16.932012 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:16.932145 (+ 133us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:16.933079 (+ 934us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:16.933084 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:16.933126 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:16.933771 (+ 645us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.933783 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.935649 (+ 1866us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.935656 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:16.937605 (+ 1949us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:16.937618 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.937781 (+ 163us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.937807 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:56:16.937880 (+ 73us) client_negotiation.cc:240] Negotiation successful
1027 14:56:16.938002 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12954,"thread_start_us":9783,"threads_started":1}
I20251027 14:56:16.939548 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.919517 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55824 (local address 127.0.104.3:46815)
1027 14:56:16.930958 (+ 11441us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:16.930967 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:16.932230 (+ 1263us) server_negotiation.cc:408] Connection header received
1027 14:56:16.932330 (+ 100us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:16.932337 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:16.932584 (+ 247us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:16.932813 (+ 229us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:16.933969 (+ 1156us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.935483 (+ 1514us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.937758 (+ 2275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.938858 (+ 1100us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.939004 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:16.939103 (+ 99us) server_negotiation.cc:300] Negotiation successful
1027 14:56:16.939282 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11301,"thread_start_us":11161,"threads_started":1}
I20251027 14:56:16.939584 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.918447 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59916 (local address 127.0.104.1:45503)
1027 14:56:16.930528 (+ 12081us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:16.930539 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:16.930582 (+ 43us) server_negotiation.cc:408] Connection header received
1027 14:56:16.930698 (+ 116us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:16.930708 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:16.931002 (+ 294us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:16.931244 (+ 242us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:16.932610 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.934249 (+ 1639us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.937731 (+ 3482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.938840 (+ 1109us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.939013 (+ 173us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:16.939134 (+ 121us) server_negotiation.cc:300] Negotiation successful
1027 14:56:16.939308 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11946,"thread_start_us":11843,"threads_started":1}
I20251027 14:56:16.939623 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:16.919188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60268 (local address 127.0.104.2:35709)
1027 14:56:16.931355 (+ 12167us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:16.931365 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:16.931394 (+ 29us) server_negotiation.cc:408] Connection header received
1027 14:56:16.931491 (+ 97us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:16.931500 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:16.931735 (+ 235us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:16.931958 (+ 223us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:16.932926 (+ 968us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.934663 (+ 1737us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:16.937669 (+ 3006us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:16.938815 (+ 1146us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:16.939020 (+ 205us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:16.939134 (+ 114us) server_negotiation.cc:300] Negotiation successful
1027 14:56:16.939308 (+ 174us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12022,"thread_start_us":11902,"threads_started":1}
I20251027 14:56:16.943636 19205 tablet_service.cc:1505] Processing CreateTablet for tablet 747db797ff034542bf0eae1b81fbfffd (DEFAULT_TABLE table=test-table [id=7693a810fbe94670a2cc12f077fabd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:16.943822 19074 tablet_service.cc:1505] Processing CreateTablet for tablet 747db797ff034542bf0eae1b81fbfffd (DEFAULT_TABLE table=test-table [id=7693a810fbe94670a2cc12f077fabd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:16.943849 19335 tablet_service.cc:1505] Processing CreateTablet for tablet 747db797ff034542bf0eae1b81fbfffd (DEFAULT_TABLE table=test-table [id=7693a810fbe94670a2cc12f077fabd8c]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:16.945214 19205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747db797ff034542bf0eae1b81fbfffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:16.945215 19335 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747db797ff034542bf0eae1b81fbfffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:16.945232 19074 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 747db797ff034542bf0eae1b81fbfffd. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:16.955863 19418 tablet_bootstrap.cc:492] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Bootstrap starting.
I20251027 14:56:16.958906 19419 tablet_bootstrap.cc:492] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Bootstrap starting.
I20251027 14:56:16.959740 19420 tablet_bootstrap.cc:492] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Bootstrap starting.
I20251027 14:56:16.961076 19418 tablet_bootstrap.cc:654] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:16.962442 19418 log.cc:826] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:16.962585 19419 tablet_bootstrap.cc:654] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:16.963147 19420 tablet_bootstrap.cc:654] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:16.963797 19419 log.cc:826] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:16.964424 19420 log.cc:826] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:16.966418 19418 tablet_bootstrap.cc:492] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: No bootstrap required, opened a new log
I20251027 14:56:16.966744 19418 ts_tablet_manager.cc:1403] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20251027 14:56:16.969172 19420 tablet_bootstrap.cc:492] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: No bootstrap required, opened a new log
I20251027 14:56:16.969434 19420 ts_tablet_manager.cc:1403] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Time spent bootstrapping tablet: real 0.010s user 0.005s sys 0.004s
I20251027 14:56:16.969449 19419 tablet_bootstrap.cc:492] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: No bootstrap required, opened a new log
I20251027 14:56:16.969736 19419 ts_tablet_manager.cc:1403] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Time spent bootstrapping tablet: real 0.011s user 0.008s sys 0.000s
I20251027 14:56:16.974020 19418 raft_consensus.cc:359] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:16.974469 19418 raft_consensus.cc:385] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:16.974581 19418 raft_consensus.cc:740] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 22fd428d276b4cfb84277345f743baff, State: Initialized, Role: FOLLOWER
I20251027 14:56:16.974834 19420 raft_consensus.cc:359] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:16.975195 19420 raft_consensus.cc:385] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:16.975296 19420 raft_consensus.cc:740] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 13429566430b46fa95bd3a625cbeab39, State: Initialized, Role: FOLLOWER
I20251027 14:56:16.975299 19418 consensus_queue.cc:260] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:16.975867 19420 consensus_queue.cc:260] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:16.976727 19270 heartbeater.cc:499] Master 127.0.104.62:35153 was elected leader, sending a full tablet report...
I20251027 14:56:16.977300 19419 raft_consensus.cc:359] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:16.977710 19419 raft_consensus.cc:385] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:16.977818 19419 raft_consensus.cc:740] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 855bbee394c6416e9a2b49c5238faf2f, State: Initialized, Role: FOLLOWER
I20251027 14:56:16.978433 19419 consensus_queue.cc:260] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:16.979188 19418 ts_tablet_manager.cc:1434] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Time spent starting tablet: real 0.012s user 0.001s sys 0.009s
I20251027 14:56:16.980219 19400 heartbeater.cc:499] Master 127.0.104.62:35153 was elected leader, sending a full tablet report...
I20251027 14:56:16.980861 19420 ts_tablet_manager.cc:1434] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Time spent starting tablet: real 0.011s user 0.008s sys 0.004s
I20251027 14:56:16.982460 19139 heartbeater.cc:499] Master 127.0.104.62:35153 was elected leader, sending a full tablet report...
I20251027 14:56:16.983055 19419 ts_tablet_manager.cc:1434] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Time spent starting tablet: real 0.013s user 0.009s sys 0.004s
W20251027 14:56:16.989861 19140 tablet.cc:2378] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:17.121516 19401 tablet.cc:2378] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:17.178941 19271 tablet.cc:2378] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:56:17.249203 19426 raft_consensus.cc:493] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:17.249446 19426 raft_consensus.cc:515] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:17.251217 19426 leader_election.cc:290] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 22fd428d276b4cfb84277345f743baff (127.0.104.2:35709), 13429566430b46fa95bd3a625cbeab39 (127.0.104.3:46815)
I20251027 14:56:17.257220 19427 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.251711 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:46815 (local address 127.0.104.1:53277)
1027 14:56:17.252502 (+ 791us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.252518 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:17.252669 (+ 151us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:17.252997 (+ 328us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:17.253001 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:17.253033 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:17.254018 (+ 985us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:17.254030 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.255254 (+ 1224us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.255258 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:17.256487 (+ 1229us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:17.256500 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.256689 (+ 189us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.256721 (+ 32us) client_negotiation.cc:769] Sending connection context
1027 14:56:17.256790 (+ 69us) client_negotiation.cc:240] Negotiation successful
1027 14:56:17.256906 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":692,"thread_start_us":452,"threads_started":1}
I20251027 14:56:17.257265 19428 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.251733 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:35709 (local address 127.0.104.1:38171)
1027 14:56:17.252564 (+ 831us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.252578 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:17.252722 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:17.253027 (+ 305us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:17.253032 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:17.253056 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:17.253976 (+ 920us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:17.253987 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.255212 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.255216 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:17.256454 (+ 1238us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:17.256466 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.256633 (+ 167us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.256666 (+ 33us) client_negotiation.cc:769] Sending connection context
1027 14:56:17.256737 (+ 71us) client_negotiation.cc:240] Negotiation successful
1027 14:56:17.256863 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":751,"spinlock_wait_cycles":29440,"thread_start_us":500,"threads_started":1}
I20251027 14:56:17.258013 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.251739 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:53277 (local address 127.0.104.3:46815)
1027 14:56:17.251931 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:17.251938 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:17.252652 (+ 714us) server_negotiation.cc:408] Connection header received
1027 14:56:17.252821 (+ 169us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:17.252825 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:17.252888 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:17.252962 (+ 74us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:17.254181 (+ 1219us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.255078 (+ 897us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.256632 (+ 1554us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.257599 (+ 967us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.257650 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:17.257724 (+ 74us) server_negotiation.cc:300] Negotiation successful
1027 14:56:17.257845 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:56:17.258069 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.251742 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:38171 (local address 127.0.104.2:35709)
1027 14:56:17.251945 (+ 203us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:17.251952 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:17.252684 (+ 732us) server_negotiation.cc:408] Connection header received
1027 14:56:17.252849 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:17.252854 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:17.252919 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:17.252996 (+ 77us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:17.254138 (+ 1142us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.255042 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.256597 (+ 1555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.257636 (+ 1039us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.257686 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:17.257759 (+ 73us) server_negotiation.cc:300] Negotiation successful
1027 14:56:17.257895 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:56:17.259016 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "747db797ff034542bf0eae1b81fbfffd" candidate_uuid: "855bbee394c6416e9a2b49c5238faf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fd428d276b4cfb84277345f743baff" is_pre_election: true
I20251027 14:56:17.259020 19355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "747db797ff034542bf0eae1b81fbfffd" candidate_uuid: "855bbee394c6416e9a2b49c5238faf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13429566430b46fa95bd3a625cbeab39" is_pre_election: true
I20251027 14:56:17.259547 19355 raft_consensus.cc:2468] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 855bbee394c6416e9a2b49c5238faf2f in term 0.
I20251027 14:56:17.259547 19225 raft_consensus.cc:2468] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 855bbee394c6416e9a2b49c5238faf2f in term 0.
I20251027 14:56:17.260329 19030 leader_election.cc:304] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 13429566430b46fa95bd3a625cbeab39, 22fd428d276b4cfb84277345f743baff, 855bbee394c6416e9a2b49c5238faf2f; no voters:
I20251027 14:56:17.260774 19426 raft_consensus.cc:2804] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:56:17.260947 19426 raft_consensus.cc:493] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:56:17.261049 19426 raft_consensus.cc:3060] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:17.263249 19426 raft_consensus.cc:515] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:17.263847 19426 leader_election.cc:290] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [CANDIDATE]: Term 1 election: Requested vote from peers 22fd428d276b4cfb84277345f743baff (127.0.104.2:35709), 13429566430b46fa95bd3a625cbeab39 (127.0.104.3:46815)
I20251027 14:56:17.264240 19225 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "747db797ff034542bf0eae1b81fbfffd" candidate_uuid: "855bbee394c6416e9a2b49c5238faf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "22fd428d276b4cfb84277345f743baff"
I20251027 14:56:17.264335 19355 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "747db797ff034542bf0eae1b81fbfffd" candidate_uuid: "855bbee394c6416e9a2b49c5238faf2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "13429566430b46fa95bd3a625cbeab39"
I20251027 14:56:17.264430 19225 raft_consensus.cc:3060] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:17.264532 19355 raft_consensus.cc:3060] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:17.266721 19355 raft_consensus.cc:2468] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855bbee394c6416e9a2b49c5238faf2f in term 1.
I20251027 14:56:17.266731 19225 raft_consensus.cc:2468] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 855bbee394c6416e9a2b49c5238faf2f in term 1.
I20251027 14:56:17.267109 19027 leader_election.cc:304] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 13429566430b46fa95bd3a625cbeab39, 855bbee394c6416e9a2b49c5238faf2f; no voters:
I20251027 14:56:17.267377 19426 raft_consensus.cc:2804] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:17.267812 19426 raft_consensus.cc:697] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 1 LEADER]: Becoming Leader. State: Replica: 855bbee394c6416e9a2b49c5238faf2f, State: Running, Role: LEADER
I20251027 14:56:17.268242 19426 consensus_queue.cc:237] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } }
I20251027 14:56:17.273330 18951 catalog_manager.cc:5649] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 855bbee394c6416e9a2b49c5238faf2f (127.0.104.1). New cstate: current_term: 1 leader_uuid: "855bbee394c6416e9a2b49c5238faf2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "855bbee394c6416e9a2b49c5238faf2f" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 45503 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 } health_report { overall_health: UNKNOWN } } }
I20251027 14:56:17.312538 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.307164 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59920 (local address 127.0.104.1:45503)
1027 14:56:17.307352 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:17.307358 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:17.307404 (+ 46us) server_negotiation.cc:408] Connection header received
1027 14:56:17.307667 (+ 263us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:17.307672 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:17.307755 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:17.307909 (+ 154us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:56:17.308505 (+ 596us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.309461 (+ 956us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.310769 (+ 1308us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.311204 (+ 435us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.311366 (+ 162us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:56:17.311919 (+ 553us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:56:17.312057 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:17.312157 (+ 100us) server_negotiation.cc:300] Negotiation successful
1027 14:56:17.312263 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251027 14:56:17.320792 19225 raft_consensus.cc:1275] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 1 FOLLOWER]: Refusing update from remote peer 855bbee394c6416e9a2b49c5238faf2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:17.320793 19355 raft_consensus.cc:1275] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 1 FOLLOWER]: Refusing update from remote peer 855bbee394c6416e9a2b49c5238faf2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:17.321775 19429 consensus_queue.cc:1048] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "22fd428d276b4cfb84277345f743baff" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:17.322037 19426 consensus_queue.cc:1048] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "13429566430b46fa95bd3a625cbeab39" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 46815 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:17.348639 18952 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:42252:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:17.348887 18952 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:42252:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:17.352849 18952 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 70aa9b77c3bb4fff94b58f9c0823f64d: Sending DeleteTablet for 3 replicas of tablet 747db797ff034542bf0eae1b81fbfffd
I20251027 14:56:17.353972 19074 tablet_service.cc:1552] Processing DeleteTablet for tablet 747db797ff034542bf0eae1b81fbfffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:17 UTC) from {username='slave'} at 127.0.0.1:59916
I20251027 14:56:17.354092 19205 tablet_service.cc:1552] Processing DeleteTablet for tablet 747db797ff034542bf0eae1b81fbfffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:17 UTC) from {username='slave'} at 127.0.0.1:60268
I20251027 14:56:17.354427 19335 tablet_service.cc:1552] Processing DeleteTablet for tablet 747db797ff034542bf0eae1b81fbfffd with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:17 UTC) from {username='slave'} at 127.0.0.1:55824
I20251027 14:56:17.355414 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19012
I20251027 14:56:17.355449 19444 tablet_replica.cc:333] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: stopping tablet replica
I20251027 14:56:17.355971 19444 raft_consensus.cc:2243] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:17.356581 19444 raft_consensus.cc:2272] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:17.357434 19442 tablet_replica.cc:333] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: stopping tablet replica
I20251027 14:56:17.357916 19442 raft_consensus.cc:2243] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:56:17.358623 19442 raft_consensus.cc:2272] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:17.359473 19443 tablet_replica.cc:333] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: stopping tablet replica
I20251027 14:56:17.359937 19443 raft_consensus.cc:2243] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:17.360488 19443 raft_consensus.cc:2272] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:17.360908 19444 ts_tablet_manager.cc:1916] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:17.362325 19443 ts_tablet_manager.cc:1916] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:17.368341 19444 ts_tablet_manager.cc:1929] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:17.368530 19444 log.cc:1199] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/747db797ff034542bf0eae1b81fbfffd
I20251027 14:56:17.368943 19442 ts_tablet_manager.cc:1916] T 747db797ff034542bf0eae1b81fbfffd P 855bbee394c6416e9a2b49c5238faf2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:17.369040 19444 ts_tablet_manager.cc:1950] T 747db797ff034542bf0eae1b81fbfffd P 13429566430b46fa95bd3a625cbeab39: Deleting consensus metadata
I20251027 14:56:17.369463 19443 ts_tablet_manager.cc:1929] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:17.369614 19443 log.cc:1199] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_1.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/747db797ff034542bf0eae1b81fbfffd
I20251027 14:56:17.370064 19443 ts_tablet_manager.cc:1950] T 747db797ff034542bf0eae1b81fbfffd P 22fd428d276b4cfb84277345f743baff: Deleting consensus metadata
I20251027 14:56:17.371003 18937 catalog_manager.cc:4985] TS 13429566430b46fa95bd3a625cbeab39 (127.0.104.3:46815): tablet 747db797ff034542bf0eae1b81fbfffd (table test-table [id=7693a810fbe94670a2cc12f077fabd8c]) successfully deleted
I20251027 14:56:17.371877 18940 catalog_manager.cc:4985] TS 22fd428d276b4cfb84277345f743baff (127.0.104.2:35709): tablet 747db797ff034542bf0eae1b81fbfffd (table test-table [id=7693a810fbe94670a2cc12f077fabd8c]) successfully deleted
I20251027 14:56:17.372210 19413 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.357842 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59930 (local address 127.0.104.1:45503)
1027 14:56:17.358000 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:17.358007 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:17.362807 (+ 4800us) server_negotiation.cc:408] Connection header received
1027 14:56:17.363089 (+ 282us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:17.363095 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:17.363188 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:17.363264 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:17.364310 (+ 1046us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.365806 (+ 1496us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.367573 (+ 1767us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.368107 (+ 534us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.369703 (+ 1596us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:17.369749 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:17.369768 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:17.370563 (+ 795us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:17.370636 (+ 73us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:17.370673 (+ 37us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:17.370758 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:17.371730 (+ 972us) server_negotiation.cc:300] Negotiation successful
1027 14:56:17.371848 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:56:17.373368 19134 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:17.473047 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19012
I20251027 14:56:17.488583 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19143
W20251027 14:56:17.488586 18939 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:45503 (error 108)
I20251027 14:56:17.490337 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.489681 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:59946)
1027 14:56:17.490096 (+ 415us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.490176 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":319,"thread_start_us":203,"threads_started":1}
W20251027 14:56:17.490536 18939 catalog_manager.cc:4712] TS 855bbee394c6416e9a2b49c5238faf2f (127.0.104.1:45503): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 747db797ff034542bf0eae1b81fbfffd: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
I20251027 14:56:17.494994 19416 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.489384 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:60274 (local address 127.0.104.2:35709)
1027 14:56:17.489572 (+ 188us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:17.489579 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:17.489678 (+ 99us) server_negotiation.cc:408] Connection header received
1027 14:56:17.489872 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:17.489877 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:17.489962 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:17.490066 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:17.490765 (+ 699us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.491670 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.492926 (+ 1256us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.493323 (+ 397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.493687 (+ 364us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:17.493731 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:17.493745 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:17.494309 (+ 564us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:17.494352 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:17.494370 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:17.494468 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:17.494588 (+ 120us) server_negotiation.cc:300] Negotiation successful
1027 14:56:17.494694 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":76}
I20251027 14:56:17.495581 19265 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:17.543980 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.543499 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:59958)
1027 14:56:17.543730 (+ 231us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.543822 (+ 92us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":95}
I20251027 14:56:17.545055 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.544675 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:59968)
1027 14:56:17.544819 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.544898 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251027 14:56:17.582831 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19143
I20251027 14:56:17.597055 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19273
I20251027 14:56:17.603199 19417 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.597775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:55834 (local address 127.0.104.3:46815)
1027 14:56:17.597924 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:17.597928 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:17.597945 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:56:17.598089 (+ 144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:17.598094 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:17.598178 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:17.598265 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:17.598903 (+ 638us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.599791 (+ 888us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:17.601043 (+ 1252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:17.601424 (+ 381us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:17.601891 (+ 467us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:17.601931 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:17.601946 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:17.602535 (+ 589us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:17.602583 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:17.602601 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:17.602689 (+ 88us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:17.602840 (+ 151us) server_negotiation.cc:300] Negotiation successful
1027 14:56:17.602936 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:56:17.603848 19395 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:17.606192 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.605635 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:59976)
1027 14:56:17.605917 (+ 282us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.606008 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20251027 14:56:17.607501 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.607131 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:59986)
1027 14:56:17.607254 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.607333 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":29}
I20251027 14:56:17.687927 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.687388 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:60002)
1027 14:56:17.687618 (+ 230us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.687724 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":82}
I20251027 14:56:17.689105 19448 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:17.688641 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:45503 (local address 127.0.0.1:60006)
1027 14:56:17.688844 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:17.688937 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:45503: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":96}
I20251027 14:56:17.693156 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19273
I20251027 14:56:17.707343 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 18922
I20251027 14:56:17.708459 18983 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:17.816841 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 18922
2025-10-27T14:56:17Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/1 (2868 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/2
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:17 dist-test-slave-hxxt krb5kdc[19454](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:17 dist-test-slave-hxxt krb5kdc[19454](info): set up 2 sockets
Oct 27 14:56:17 dist-test-slave-hxxt krb5kdc[19454](info): commencing operation
krb5kdc: starting...
W20251027 14:56:19.712510 416 mini_kdc.cc:121] Time spent starting KDC: real 1.858s user 0.000s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:19 dist-test-slave-hxxt krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:19.991853 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40525
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:40525
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:20.319458 19470 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:20.319774 19470 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:20.319908 19470 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:20.327827 19470 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:20.327934 19470 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:20.328019 19470 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:20.328083 19470 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:20.328163 19470 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:20.338480 19470 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:40525
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:40525
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:20.340001 19470 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:20.341305 19470 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:20.349481 19475 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:20.349442 19476 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:20.350234 19478 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:20.350947 19470 server_base.cc:1047] running on GCE node
I20251027 14:56:20.351682 19470 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:20.353142 19470 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:20.354368 19470 hybrid_clock.cc:648] HybridClock initialized: now 1761576980354301 us; error 44 us; skew 500 ppm
I20251027 14:56:20.354699 19470 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:20 dist-test-slave-hxxt krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:20.357808 19470 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:56:20.359345 19470 webserver.cc:492] Webserver started at http://127.0.104.62:35579/ using document root <none> and password file <none>
I20251027 14:56:20.360044 19470 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:20.360128 19470 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:20.360517 19470 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:20.363049 19470 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "59350b9c4b134e74afc57be956c90624"
format_stamp: "Formatted at 2025-10-27 14:56:20 on dist-test-slave-hxxt"
server_key: "93a0f936053bb2df1ed9d5407a05636b"
server_key_iv: "38b926feeca9f52da869963b615d84c9"
server_key_version: "encryptionkey@0"
I20251027 14:56:20.363787 19470 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "59350b9c4b134e74afc57be956c90624"
format_stamp: "Formatted at 2025-10-27 14:56:20 on dist-test-slave-hxxt"
server_key: "93a0f936053bb2df1ed9d5407a05636b"
server_key_iv: "38b926feeca9f52da869963b615d84c9"
server_key_version: "encryptionkey@0"
I20251027 14:56:20.368997 19470 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20251027 14:56:20.372195 19485 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:20.373476 19470 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:20.373624 19470 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "59350b9c4b134e74afc57be956c90624"
format_stamp: "Formatted at 2025-10-27 14:56:20 on dist-test-slave-hxxt"
server_key: "93a0f936053bb2df1ed9d5407a05636b"
server_key_iv: "38b926feeca9f52da869963b615d84c9"
server_key_version: "encryptionkey@0"
I20251027 14:56:20.373864 19470 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:20.409478 19470 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:20.416935 19470 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:20.417196 19470 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:20.435151 19470 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:40525
I20251027 14:56:20.435163 19536 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:40525 every 8 connection(s)
I20251027 14:56:20.436635 19470 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:20.440814 19537 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:20.445266 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19470
I20251027 14:56:20.445448 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:20.445848 416 external_mini_cluster.cc:1468] Setting key b98ad31c2f1198f534f3ff6a502f4941
I20251027 14:56:20.452041 19537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624: Bootstrap starting.
I20251027 14:56:20.455781 19537 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:20.457191 19537 log.cc:826] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624: Log is configured to *not* fsync() on all Append() calls
Oct 27 14:56:20 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:20.460265 19537 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624: No bootstrap required, opened a new log
I20251027 14:56:20.465973 19537 raft_consensus.cc:359] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59350b9c4b134e74afc57be956c90624" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40525 } }
I20251027 14:56:20.466403 19537 raft_consensus.cc:385] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:20.466501 19537 raft_consensus.cc:740] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 59350b9c4b134e74afc57be956c90624, State: Initialized, Role: FOLLOWER
I20251027 14:56:20.467109 19537 consensus_queue.cc:260] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59350b9c4b134e74afc57be956c90624" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40525 } }
I20251027 14:56:20.467301 19537 raft_consensus.cc:399] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:20.467456 19537 raft_consensus.cc:493] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:20.467464 19540 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:20.447610 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:51200 (local address 127.0.104.62:40525)
1027 14:56:20.448417 (+ 807us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:20.448432 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:20.448475 (+ 43us) server_negotiation.cc:408] Connection header received
1027 14:56:20.449795 (+ 1320us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:20.449819 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:20.450257 (+ 438us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:20.450629 (+ 372us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:20.452783 (+ 2154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:20.454205 (+ 1422us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:20.455822 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:20.456439 (+ 617us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:20.460168 (+ 3729us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:20.460232 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:20.460251 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:20.460286 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:20.463349 (+ 3063us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:20.464074 (+ 725us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:20.464082 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:20.464088 (+ 6us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:20.464195 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:20.464560 (+ 365us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:20.464564 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:20.464566 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:20.465124 (+ 558us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:20.465445 (+ 321us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:20.465880 (+ 435us) server_negotiation.cc:300] Negotiation successful
1027 14:56:20.466302 (+ 422us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":587,"thread_start_us":387,"threads_started":1}
I20251027 14:56:20.467639 19537 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:20.470407 19537 raft_consensus.cc:515] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59350b9c4b134e74afc57be956c90624" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40525 } }
I20251027 14:56:20.470942 19537 leader_election.cc:304] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 59350b9c4b134e74afc57be956c90624; no voters:
I20251027 14:56:20.471495 19537 leader_election.cc:290] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:20.471616 19542 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:20.472219 19542 raft_consensus.cc:697] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [term 1 LEADER]: Becoming Leader. State: Replica: 59350b9c4b134e74afc57be956c90624, State: Running, Role: LEADER
I20251027 14:56:20.472688 19542 consensus_queue.cc:237] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59350b9c4b134e74afc57be956c90624" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40525 } }
I20251027 14:56:20.473356 19537 sys_catalog.cc:565] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:20.475164 19544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 59350b9c4b134e74afc57be956c90624. Latest consensus state: current_term: 1 leader_uuid: "59350b9c4b134e74afc57be956c90624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59350b9c4b134e74afc57be956c90624" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40525 } } }
I20251027 14:56:20.475455 19544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:20.476274 19543 sys_catalog.cc:455] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "59350b9c4b134e74afc57be956c90624" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "59350b9c4b134e74afc57be956c90624" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 40525 } } }
I20251027 14:56:20.476642 19543 sys_catalog.cc:458] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:20.477916 19547 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:20.483232 19547 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:20.492509 19547 catalog_manager.cc:1357] Generated new cluster ID: 36b85591e4de4982a1eb8d61886757e6
I20251027 14:56:20.492625 19547 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:20.506400 19547 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:20.508090 19547 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:20.519531 19547 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624: Generated new TSK 0
I20251027 14:56:20.520629 19547 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:20.629784 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40525
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:20.954066 19565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:20.954334 19565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:20.954463 19565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:20.962366 19565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:20.962455 19565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:20.962639 19565 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:56:20.972797 19565 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:40525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:20.974313 19565 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:20.975629 19565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:20.985106 19571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:20.985155 19570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:20.985309 19573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:20.986368 19565 server_base.cc:1047] running on GCE node
I20251027 14:56:20.986899 19565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:20.987733 19565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:20.988981 19565 hybrid_clock.cc:648] HybridClock initialized: now 1761576980988906 us; error 40 us; skew 500 ppm
I20251027 14:56:20.989284 19565 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:20 dist-test-slave-hxxt krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:20.992427 19565 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:56:20.993978 19565 webserver.cc:492] Webserver started at http://127.0.104.1:44747/ using document root <none> and password file <none>
I20251027 14:56:20.994668 19565 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:20.994752 19565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:20.995123 19565 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:20.997748 19565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "2c24cb05132a49aeb48e3a8c723e5857"
format_stamp: "Formatted at 2025-10-27 14:56:20 on dist-test-slave-hxxt"
server_key: "e47fbc2b4e5eb9cd2118199d82bf9ba8"
server_key_iv: "1e34f2b19540a5016c505d7b8e4705a4"
server_key_version: "encryptionkey@0"
I20251027 14:56:20.998534 19565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "2c24cb05132a49aeb48e3a8c723e5857"
format_stamp: "Formatted at 2025-10-27 14:56:20 on dist-test-slave-hxxt"
server_key: "e47fbc2b4e5eb9cd2118199d82bf9ba8"
server_key_iv: "1e34f2b19540a5016c505d7b8e4705a4"
server_key_version: "encryptionkey@0"
I20251027 14:56:21.003878 19565 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:56:21.007508 19580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:21.009012 19565 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:21.009171 19565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "2c24cb05132a49aeb48e3a8c723e5857"
format_stamp: "Formatted at 2025-10-27 14:56:20 on dist-test-slave-hxxt"
server_key: "e47fbc2b4e5eb9cd2118199d82bf9ba8"
server_key_iv: "1e34f2b19540a5016c505d7b8e4705a4"
server_key_version: "encryptionkey@0"
I20251027 14:56:21.009382 19565 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:21.029284 19565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:21.033421 19565 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:21.033681 19565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:21.034638 19565 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:21.036195 19565 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:21.036281 19565 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:21.036405 19565 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:21.036479 19565 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20251027 14:56:21.066792 19565 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:35483
I20251027 14:56:21.066835 19692 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:35483 every 8 connection(s)
I20251027 14:56:21.068150 19565 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:56:21.072698 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19565
I20251027 14:56:21.072867 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:56:21.073267 416 external_mini_cluster.cc:1468] Setting key ce559601647493e70b3233b7a895b182
Oct 27 14:56:21 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576980, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:21.087563 19695 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:21.070882 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:40359 (local address 127.0.104.62:40525)
1027 14:56:21.071328 (+ 446us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:21.071333 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:21.072321 (+ 988us) server_negotiation.cc:408] Connection header received
1027 14:56:21.074056 (+ 1735us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:21.074064 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:21.074159 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:21.074261 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:21.076461 (+ 2200us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.077430 (+ 969us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:21.078642 (+ 1212us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.079101 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:21.081925 (+ 2824us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:21.081956 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:21.081959 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:21.081991 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:21.084517 (+ 2526us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:21.085380 (+ 863us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:21.085385 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:21.085388 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:21.085475 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:21.085811 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:21.085815 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:21.085817 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:21.086066 (+ 249us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:21.086181 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:21.087090 (+ 909us) server_negotiation.cc:300] Negotiation successful
1027 14:56:21.087300 (+ 210us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":352,"thread_start_us":226,"threads_started":1}
I20251027 14:56:21.088811 19696 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:21.071340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:40525 (local address 127.0.104.1:40359)
1027 14:56:21.072116 (+ 776us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:21.072158 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:21.073670 (+ 1512us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:21.074529 (+ 859us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:21.074538 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:21.075357 (+ 819us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:21.076263 (+ 906us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:21.076283 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.077577 (+ 1294us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:21.077581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:21.078524 (+ 943us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:21.078537 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.078779 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:21.079625 (+ 846us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:21.079651 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:21.081732 (+ 2081us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:21.084666 (+ 2934us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:21.084675 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:21.084694 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:21.085260 (+ 566us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:21.085568 (+ 308us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:21.085573 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:21.085576 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:21.085710 (+ 134us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:21.086172 (+ 462us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:21.086180 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:21.086586 (+ 406us) client_negotiation.cc:769] Sending connection context
1027 14:56:21.086981 (+ 395us) client_negotiation.cc:240] Negotiation successful
1027 14:56:21.087463 (+ 482us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":502,"thread_start_us":329,"threads_started":1}
I20251027 14:56:21.090440 19693 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40525
I20251027 14:56:21.090843 19693 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:21.091818 19693 heartbeater.cc:507] Master 127.0.104.62:40525 requested a full tablet report, sending...
I20251027 14:56:21.094167 19502 ts_manager.cc:194] Registered new tserver with Master: 2c24cb05132a49aeb48e3a8c723e5857 (127.0.104.1:35483)
I20251027 14:56:21.095715 19502 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:40359
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:21.170712 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40525
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:21.489486 19701 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:21.489771 19701 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:21.489876 19701 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:21.497711 19701 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:21.497807 19701 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:21.497949 19701 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:56:21.508100 19701 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:40525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:21.509613 19701 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:21.510878 19701 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:21.519722 19707 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:21.519685 19706 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:21.520354 19709 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:21.520877 19701 server_base.cc:1047] running on GCE node
I20251027 14:56:21.521456 19701 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:21.522333 19701 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:21.523561 19701 hybrid_clock.cc:648] HybridClock initialized: now 1761576981523486 us; error 45 us; skew 500 ppm
I20251027 14:56:21.523876 19701 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:21 dist-test-slave-hxxt krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:21.526955 19701 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:56:21.528477 19701 webserver.cc:492] Webserver started at http://127.0.104.2:45843/ using document root <none> and password file <none>
I20251027 14:56:21.529194 19701 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:21.529279 19701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:21.529629 19701 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:21.532191 19701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "664f9b2ee0804925b058cd06a1bd61b7"
format_stamp: "Formatted at 2025-10-27 14:56:21 on dist-test-slave-hxxt"
server_key: "3ab4345bbb5990db5a13139a86af36e9"
server_key_iv: "c1e262d58c282435ad85d4e336576c21"
server_key_version: "encryptionkey@0"
I20251027 14:56:21.532990 19701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "664f9b2ee0804925b058cd06a1bd61b7"
format_stamp: "Formatted at 2025-10-27 14:56:21 on dist-test-slave-hxxt"
server_key: "3ab4345bbb5990db5a13139a86af36e9"
server_key_iv: "c1e262d58c282435ad85d4e336576c21"
server_key_version: "encryptionkey@0"
I20251027 14:56:21.538194 19701 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:56:21.541651 19716 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:21.542906 19701 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:56:21.543051 19701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "664f9b2ee0804925b058cd06a1bd61b7"
format_stamp: "Formatted at 2025-10-27 14:56:21 on dist-test-slave-hxxt"
server_key: "3ab4345bbb5990db5a13139a86af36e9"
server_key_iv: "c1e262d58c282435ad85d4e336576c21"
server_key_version: "encryptionkey@0"
I20251027 14:56:21.543280 19701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:21.570673 19701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:21.574842 19701 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:21.575095 19701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:21.576061 19701 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:21.577608 19701 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:21.577702 19701 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:21.577800 19701 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:21.577883 19701 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:21.607708 19701 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:35221
I20251027 14:56:21.607764 19828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:35221 every 8 connection(s)
I20251027 14:56:21.609169 19701 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:56:21.612797 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19701
I20251027 14:56:21.612970 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:56:21.613348 416 external_mini_cluster.cc:1468] Setting key 109e1e719173baf1703939b0ac851cc3
Oct 27 14:56:21 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576981, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:21.628542 19831 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:21.611863 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:45269 (local address 127.0.104.62:40525)
1027 14:56:21.612330 (+ 467us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:21.612335 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:21.613806 (+ 1471us) server_negotiation.cc:408] Connection header received
1027 14:56:21.615215 (+ 1409us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:21.615221 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:21.615319 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:21.615418 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:21.617187 (+ 1769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.618085 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:21.619178 (+ 1093us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.619641 (+ 463us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:21.622801 (+ 3160us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:21.622842 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:21.622846 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:21.622879 (+ 33us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:21.625618 (+ 2739us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:21.626322 (+ 704us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:21.626326 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:21.626329 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:21.626422 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:21.626750 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:21.626755 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:21.626757 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:21.627010 (+ 253us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:21.627150 (+ 140us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:21.627891 (+ 741us) server_negotiation.cc:300] Negotiation successful
1027 14:56:21.628150 (+ 259us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":338,"thread_start_us":226,"threads_started":1}
I20251027 14:56:21.629456 19832 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:21.612482 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:40525 (local address 127.0.104.2:45269)
1027 14:56:21.613278 (+ 796us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:21.613328 (+ 50us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:21.614904 (+ 1576us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:21.615679 (+ 775us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:21.615688 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:21.616255 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:21.617005 (+ 750us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:21.617024 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.618236 (+ 1212us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:21.618240 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:21.619030 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:21.619041 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:21.619322 (+ 281us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:21.620203 (+ 881us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:21.620228 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:21.622581 (+ 2353us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:21.625768 (+ 3187us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:21.625777 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:21.625796 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:21.626195 (+ 399us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:21.626521 (+ 326us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:21.626525 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:21.626528 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:21.626643 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:21.627129 (+ 486us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:21.627137 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:21.627472 (+ 335us) client_negotiation.cc:769] Sending connection context
1027 14:56:21.627778 (+ 306us) client_negotiation.cc:240] Negotiation successful
1027 14:56:21.628267 (+ 489us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":343,"threads_started":1}
I20251027 14:56:21.630858 19829 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40525
I20251027 14:56:21.631222 19829 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:21.632000 19829 heartbeater.cc:507] Master 127.0.104.62:40525 requested a full tablet report, sending...
I20251027 14:56:21.633596 19502 ts_manager.cc:194] Registered new tserver with Master: 664f9b2ee0804925b058cd06a1bd61b7 (127.0.104.2:35221)
I20251027 14:56:21.634775 19502 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:45269
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:21.709537 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:40525
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37277
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:22.033198 19837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:22.033493 19837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:22.033587 19837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:22.041596 19837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:22.041693 19837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:22.041838 19837 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:56:22.052174 19837 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37277
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:40525
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:22.053745 19837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:22.055018 19837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:22.064056 19843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:22.064117 19842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:22.064317 19845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:22.065104 19837 server_base.cc:1047] running on GCE node
I20251027 14:56:22.065723 19837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:22.066789 19837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:22.068038 19837 hybrid_clock.cc:648] HybridClock initialized: now 1761576982067975 us; error 33 us; skew 500 ppm
I20251027 14:56:22.068405 19837 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:22 dist-test-slave-hxxt krb5kdc[19454](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:22.072312 19837 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:56:22.073915 19837 webserver.cc:492] Webserver started at http://127.0.104.3:35987/ using document root <none> and password file <none>
I20251027 14:56:22.074632 19837 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:22.074713 19837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:22.075071 19837 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:22.077680 19837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "dd9743c815ff4497978b86557a625bd3"
format_stamp: "Formatted at 2025-10-27 14:56:22 on dist-test-slave-hxxt"
server_key: "b6760f3139b9ec6158339a6f91ce554e"
server_key_iv: "53663024f705df20bfdeae426f82c377"
server_key_version: "encryptionkey@0"
I20251027 14:56:22.078451 19837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "dd9743c815ff4497978b86557a625bd3"
format_stamp: "Formatted at 2025-10-27 14:56:22 on dist-test-slave-hxxt"
server_key: "b6760f3139b9ec6158339a6f91ce554e"
server_key_iv: "53663024f705df20bfdeae426f82c377"
server_key_version: "encryptionkey@0"
I20251027 14:56:22.083925 19837 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20251027 14:56:22.087574 19852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:22.089078 19837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:22.089232 19837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "dd9743c815ff4497978b86557a625bd3"
format_stamp: "Formatted at 2025-10-27 14:56:22 on dist-test-slave-hxxt"
server_key: "b6760f3139b9ec6158339a6f91ce554e"
server_key_iv: "53663024f705df20bfdeae426f82c377"
server_key_version: "encryptionkey@0"
I20251027 14:56:22.089444 19837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:22.098793 19693 heartbeater.cc:499] Master 127.0.104.62:40525 was elected leader, sending a full tablet report...
I20251027 14:56:22.109759 19837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:22.113781 19837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:22.114030 19837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:22.114976 19837 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:22.116575 19837 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:22.116669 19837 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:22.116777 19837 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:22.116848 19837 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:22.147691 19837 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34849
I20251027 14:56:22.147769 19964 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34849 every 8 connection(s)
I20251027 14:56:22.149142 19837 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:56:22.152870 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 19837
I20251027 14:56:22.153074 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:56:22.153503 416 external_mini_cluster.cc:1468] Setting key 9c5c251b1393c64b7219b045bbe47f64
Oct 27 14:56:22 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576982, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:22.167553 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.151949 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:38029 (local address 127.0.104.62:40525)
1027 14:56:22.152411 (+ 462us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.152417 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.153528 (+ 1111us) server_negotiation.cc:408] Connection header received
1027 14:56:22.155213 (+ 1685us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.155219 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.155323 (+ 104us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.155419 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:22.157289 (+ 1870us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.158259 (+ 970us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.159290 (+ 1031us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.159624 (+ 334us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.162290 (+ 2666us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:22.162331 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:22.162334 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:22.162365 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:22.164762 (+ 2397us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.165426 (+ 664us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.165431 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.165433 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.165503 (+ 70us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.165822 (+ 319us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.165826 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.165828 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.166052 (+ 224us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:22.166179 (+ 127us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.166999 (+ 820us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.167214 (+ 215us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":336,"thread_start_us":229,"threads_started":1}
I20251027 14:56:22.168576 19968 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.152514 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:40525 (local address 127.0.104.3:38029)
1027 14:56:22.153331 (+ 817us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:22.153379 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:22.154932 (+ 1553us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:22.155684 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:22.155694 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:22.156397 (+ 703us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:22.157129 (+ 732us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.157149 (+ 20us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.158391 (+ 1242us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.158395 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:22.159172 (+ 777us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.159182 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.159382 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.160027 (+ 645us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:22.160051 (+ 24us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:22.162111 (+ 2060us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:22.164893 (+ 2782us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:22.164902 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:22.164920 (+ 18us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:22.165308 (+ 388us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:22.165601 (+ 293us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:22.165606 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:22.165610 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:22.165717 (+ 107us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:22.166165 (+ 448us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:22.166173 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:22.166486 (+ 313us) client_negotiation.cc:769] Sending connection context
1027 14:56:22.166857 (+ 371us) client_negotiation.cc:240] Negotiation successful
1027 14:56:22.167302 (+ 445us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":551,"thread_start_us":362,"threads_started":1}
I20251027 14:56:22.169895 19965 heartbeater.cc:344] Connected to a master server at 127.0.104.62:40525
I20251027 14:56:22.170265 19965 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:22.171033 19965 heartbeater.cc:507] Master 127.0.104.62:40525 requested a full tablet report, sending...
I20251027 14:56:22.172430 19502 ts_manager.cc:194] Registered new tserver with Master: dd9743c815ff4497978b86557a625bd3 (127.0.104.3:34849)
I20251027 14:56:22.173202 19502 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:38029
I20251027 14:56:22.181464 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:56:22.198266 19967 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.185861 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59492 (local address 127.0.104.62:40525)
1027 14:56:22.186044 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.186052 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.186364 (+ 312us) server_negotiation.cc:408] Connection header received
1027 14:56:22.186529 (+ 165us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.186533 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.186598 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.186681 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:22.188304 (+ 1623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.189268 (+ 964us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.190495 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.190990 (+ 495us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.193022 (+ 2032us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:22.193066 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:22.193071 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:22.193110 (+ 39us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:22.195754 (+ 2644us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.196504 (+ 750us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.196509 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.196511 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.196582 (+ 71us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.196930 (+ 348us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.196934 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.196936 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.197195 (+ 259us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:22.197501 (+ 306us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.197801 (+ 300us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.197980 (+ 179us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":49}
I20251027 14:56:22.202029 19502 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59492:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:56:22.206629 19502 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:56:22.228965 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.219261 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42190 (local address 127.0.104.1:35483)
1027 14:56:22.219803 (+ 542us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.219812 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.219845 (+ 33us) server_negotiation.cc:408] Connection header received
1027 14:56:22.220113 (+ 268us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.220120 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.220314 (+ 194us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.220558 (+ 244us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:22.221737 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.224211 (+ 2474us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.226780 (+ 2569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.227847 (+ 1067us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.227986 (+ 139us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.228498 (+ 512us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.228689 (+ 191us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":415,"thread_start_us":304,"threads_started":1}
I20251027 14:56:22.229768 19978 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.218993 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:35483 (local address 127.0.0.1:42190)
1027 14:56:22.219620 (+ 627us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:22.219660 (+ 40us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:22.219857 (+ 197us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:22.220545 (+ 688us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:22.220554 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:22.220585 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:22.221555 (+ 970us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.221573 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.224485 (+ 2912us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.224492 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:22.226576 (+ 2084us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.226594 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.228097 (+ 1503us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.228155 (+ 58us) client_negotiation.cc:769] Sending connection context
1027 14:56:22.229232 (+ 1077us) client_negotiation.cc:240] Negotiation successful
1027 14:56:22.229429 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":397,"thread_start_us":885,"threads_started":1}
I20251027 14:56:22.231276 19981 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.220624 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34849 (local address 127.0.0.1:33920)
1027 14:56:22.221370 (+ 746us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:22.221390 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:22.221510 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:22.225474 (+ 3964us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:22.225479 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:22.225510 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:22.226184 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.226198 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.228449 (+ 2251us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.228453 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:22.229759 (+ 1306us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.229771 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.229943 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.229964 (+ 21us) client_negotiation.cc:769] Sending connection context
1027 14:56:22.230028 (+ 64us) client_negotiation.cc:240] Negotiation successful
1027 14:56:22.230146 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":636,"thread_start_us":4244,"threads_started":1}
I20251027 14:56:22.231603 19980 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.220795 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:35221 (local address 127.0.0.1:49674)
1027 14:56:22.225172 (+ 4377us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:22.225193 (+ 21us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:22.225331 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:22.226202 (+ 871us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:22.226207 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:22.226237 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:22.226927 (+ 690us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.226939 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.228495 (+ 1556us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.228500 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:22.230202 (+ 1702us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.230216 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.230372 (+ 156us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.230394 (+ 22us) client_negotiation.cc:769] Sending connection context
1027 14:56:22.230453 (+ 59us) client_negotiation.cc:240] Negotiation successful
1027 14:56:22.230563 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":4280,"thread_start_us":1286,"threads_started":1}
I20251027 14:56:22.232048 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.221317 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33920 (local address 127.0.104.3:34849)
1027 14:56:22.224872 (+ 3555us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.224882 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.224914 (+ 32us) server_negotiation.cc:408] Connection header received
1027 14:56:22.225024 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.225031 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.225261 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.225510 (+ 249us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:22.226370 (+ 860us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.228200 (+ 1830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.230058 (+ 1858us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.231154 (+ 1096us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.231290 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.231581 (+ 291us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.231862 (+ 281us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3396,"thread_start_us":3232,"threads_started":1}
I20251027 14:56:22.233137 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.221668 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49674 (local address 127.0.104.2:35221)
1027 14:56:22.222143 (+ 475us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.222151 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.225333 (+ 3182us) server_negotiation.cc:408] Connection header received
1027 14:56:22.225476 (+ 143us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.225484 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.225711 (+ 227us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.225917 (+ 206us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:22.227082 (+ 1165us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.228236 (+ 1154us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.230643 (+ 2407us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.232497 (+ 1854us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.232614 (+ 117us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.232743 (+ 129us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.232898 (+ 155us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":347,"thread_start_us":244,"threads_started":1}
I20251027 14:56:22.234520 19628 tablet_service.cc:1505] Processing CreateTablet for tablet a0d7118b44cc42258c57ae5780c0516d (DEFAULT_TABLE table=test-table [id=13d0d5f1d2714a2d8dddad425bce99ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:22.235760 19628 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d7118b44cc42258c57ae5780c0516d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:22.235654 19900 tablet_service.cc:1505] Processing CreateTablet for tablet a0d7118b44cc42258c57ae5780c0516d (DEFAULT_TABLE table=test-table [id=13d0d5f1d2714a2d8dddad425bce99ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:22.237118 19900 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d7118b44cc42258c57ae5780c0516d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:22.237162 19764 tablet_service.cc:1505] Processing CreateTablet for tablet a0d7118b44cc42258c57ae5780c0516d (DEFAULT_TABLE table=test-table [id=13d0d5f1d2714a2d8dddad425bce99ad]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:22.238494 19764 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a0d7118b44cc42258c57ae5780c0516d. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:22.246601 19984 tablet_bootstrap.cc:492] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Bootstrap starting.
I20251027 14:56:22.249836 19984 tablet_bootstrap.cc:654] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:22.250867 19984 log.cc:826] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:22.251814 19985 tablet_bootstrap.cc:492] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Bootstrap starting.
I20251027 14:56:22.253790 19986 tablet_bootstrap.cc:492] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Bootstrap starting.
I20251027 14:56:22.255913 19985 tablet_bootstrap.cc:654] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:22.256678 19986 tablet_bootstrap.cc:654] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:22.257019 19984 tablet_bootstrap.cc:492] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: No bootstrap required, opened a new log
I20251027 14:56:22.257246 19985 log.cc:826] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:22.257299 19984 ts_tablet_manager.cc:1403] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20251027 14:56:22.257663 19986 log.cc:826] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:22.260426 19985 tablet_bootstrap.cc:492] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: No bootstrap required, opened a new log
I20251027 14:56:22.260725 19986 tablet_bootstrap.cc:492] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: No bootstrap required, opened a new log
I20251027 14:56:22.260721 19985 ts_tablet_manager.cc:1403] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.003s
I20251027 14:56:22.260967 19986 ts_tablet_manager.cc:1403] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Time spent bootstrapping tablet: real 0.007s user 0.006s sys 0.000s
I20251027 14:56:22.263334 19984 raft_consensus.cc:359] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.263660 19984 raft_consensus.cc:385] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:22.263754 19984 raft_consensus.cc:740] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c24cb05132a49aeb48e3a8c723e5857, State: Initialized, Role: FOLLOWER
I20251027 14:56:22.264340 19984 consensus_queue.cc:260] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.265969 19984 ts_tablet_manager.cc:1434] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Time spent starting tablet: real 0.009s user 0.008s sys 0.000s
I20251027 14:56:22.267882 19986 raft_consensus.cc:359] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.267882 19985 raft_consensus.cc:359] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.268256 19986 raft_consensus.cc:385] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:22.268257 19985 raft_consensus.cc:385] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:22.268363 19985 raft_consensus.cc:740] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dd9743c815ff4497978b86557a625bd3, State: Initialized, Role: FOLLOWER
I20251027 14:56:22.268360 19986 raft_consensus.cc:740] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 664f9b2ee0804925b058cd06a1bd61b7, State: Initialized, Role: FOLLOWER
I20251027 14:56:22.269026 19986 consensus_queue.cc:260] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.269026 19985 consensus_queue.cc:260] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.270473 19829 heartbeater.cc:499] Master 127.0.104.62:40525 was elected leader, sending a full tablet report...
I20251027 14:56:22.270643 19985 ts_tablet_manager.cc:1434] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Time spent starting tablet: real 0.010s user 0.005s sys 0.004s
I20251027 14:56:22.270917 19986 ts_tablet_manager.cc:1434] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Time spent starting tablet: real 0.010s user 0.006s sys 0.004s
I20251027 14:56:22.271958 19965 heartbeater.cc:499] Master 127.0.104.62:40525 was elected leader, sending a full tablet report...
W20251027 14:56:22.320536 19694 tablet.cc:2378] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:22.361171 19830 tablet.cc:2378] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:56:22.377753 19990 raft_consensus.cc:493] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:22.378050 19990 raft_consensus.cc:515] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.379928 19990 leader_election.cc:290] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 664f9b2ee0804925b058cd06a1bd61b7 (127.0.104.2:35221), dd9743c815ff4497978b86557a625bd3 (127.0.104.3:34849)
I20251027 14:56:22.385499 19993 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.380273 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:35221 (local address 127.0.104.1:45147)
1027 14:56:22.381126 (+ 853us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:22.381143 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:22.381271 (+ 128us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:22.381566 (+ 295us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:22.381570 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:22.381619 (+ 49us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:22.382306 (+ 687us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.382317 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.383577 (+ 1260us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.383581 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:22.384834 (+ 1253us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.384846 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.384981 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.385005 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:56:22.385053 (+ 48us) client_negotiation.cc:240] Negotiation successful
1027 14:56:22.385143 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":777,"thread_start_us":496,"threads_started":1}
I20251027 14:56:22.385528 19994 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.380233 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34849 (local address 127.0.104.1:35225)
1027 14:56:22.381068 (+ 835us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:22.381083 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:22.381203 (+ 120us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:22.381537 (+ 334us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:22.381542 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:22.381565 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:22.382290 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.382300 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.383525 (+ 1225us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.383530 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:22.384831 (+ 1301us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:22.384844 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.385021 (+ 177us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.385047 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:56:22.385112 (+ 65us) client_negotiation.cc:240] Negotiation successful
1027 14:56:22.385225 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":737,"thread_start_us":606,"threads_started":1}
I20251027 14:56:22.386590 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.380596 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:35225 (local address 127.0.104.3:34849)
1027 14:56:22.380778 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.380784 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.381170 (+ 386us) server_negotiation.cc:408] Connection header received
1027 14:56:22.381361 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.381368 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.381433 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.381498 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:22.382427 (+ 929us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.383326 (+ 899us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.384974 (+ 1648us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.386102 (+ 1128us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.386153 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.386228 (+ 75us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.386352 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:56:22.387220 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.380292 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:45147 (local address 127.0.104.2:35221)
1027 14:56:22.380486 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.380493 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.381243 (+ 750us) server_negotiation.cc:408] Connection header received
1027 14:56:22.381395 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.381399 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.381456 (+ 57us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.381539 (+ 83us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:22.382448 (+ 909us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.383329 (+ 881us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.385855 (+ 2526us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.386884 (+ 1029us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.386922 (+ 38us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.386983 (+ 61us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.387068 (+ 85us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251027 14:56:22.387579 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0d7118b44cc42258c57ae5780c0516d" candidate_uuid: "2c24cb05132a49aeb48e3a8c723e5857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9743c815ff4497978b86557a625bd3" is_pre_election: true
I20251027 14:56:22.388022 19920 raft_consensus.cc:2468] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c24cb05132a49aeb48e3a8c723e5857 in term 0.
I20251027 14:56:22.388000 19784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0d7118b44cc42258c57ae5780c0516d" candidate_uuid: "2c24cb05132a49aeb48e3a8c723e5857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664f9b2ee0804925b058cd06a1bd61b7" is_pre_election: true
I20251027 14:56:22.388460 19784 raft_consensus.cc:2468] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2c24cb05132a49aeb48e3a8c723e5857 in term 0.
I20251027 14:56:22.388684 19584 leader_election.cc:304] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c24cb05132a49aeb48e3a8c723e5857, dd9743c815ff4497978b86557a625bd3; no voters:
I20251027 14:56:22.389091 19990 raft_consensus.cc:2804] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:56:22.389233 19990 raft_consensus.cc:493] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:56:22.389326 19990 raft_consensus.cc:3060] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:22.391571 19990 raft_consensus.cc:515] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
I20251027 14:56:22.392143 19990 leader_election.cc:290] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [CANDIDATE]: Term 1 election: Requested vote from peers 664f9b2ee0804925b058cd06a1bd61b7 (127.0.104.2:35221), dd9743c815ff4497978b86557a625bd3 (127.0.104.3:34849)
I20251027 14:56:22.392561 19784 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0d7118b44cc42258c57ae5780c0516d" candidate_uuid: "2c24cb05132a49aeb48e3a8c723e5857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "664f9b2ee0804925b058cd06a1bd61b7"
I20251027 14:56:22.392596 19920 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "a0d7118b44cc42258c57ae5780c0516d" candidate_uuid: "2c24cb05132a49aeb48e3a8c723e5857" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "dd9743c815ff4497978b86557a625bd3"
I20251027 14:56:22.392731 19784 raft_consensus.cc:3060] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:22.392756 19920 raft_consensus.cc:3060] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:22.395076 19784 raft_consensus.cc:2468] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c24cb05132a49aeb48e3a8c723e5857 in term 1.
I20251027 14:56:22.395123 19920 raft_consensus.cc:2468] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2c24cb05132a49aeb48e3a8c723e5857 in term 1.
I20251027 14:56:22.395483 19581 leader_election.cc:304] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c24cb05132a49aeb48e3a8c723e5857, 664f9b2ee0804925b058cd06a1bd61b7; no voters:
I20251027 14:56:22.395753 19990 raft_consensus.cc:2804] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:22.396152 19990 raft_consensus.cc:697] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 1 LEADER]: Becoming Leader. State: Replica: 2c24cb05132a49aeb48e3a8c723e5857, State: Running, Role: LEADER
I20251027 14:56:22.396682 19990 consensus_queue.cc:237] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } }
W20251027 14:56:22.400934 19966 tablet.cc:2378] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:56:22.404408 19501 catalog_manager.cc:5649] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 reported cstate change: term changed from 0 to 1, leader changed from <none> to 2c24cb05132a49aeb48e3a8c723e5857 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "2c24cb05132a49aeb48e3a8c723e5857" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c24cb05132a49aeb48e3a8c723e5857" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 35483 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 } health_report { overall_health: UNKNOWN } } }
I20251027 14:56:22.426558 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.421130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42196 (local address 127.0.104.1:35483)
1027 14:56:22.421278 (+ 148us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.421283 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.421397 (+ 114us) server_negotiation.cc:408] Connection header received
1027 14:56:22.421653 (+ 256us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.421659 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.421743 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.421871 (+ 128us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:56:22.422396 (+ 525us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.423324 (+ 928us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.424666 (+ 1342us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.425012 (+ 346us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.425442 (+ 430us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:56:22.425990 (+ 548us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:56:22.426136 (+ 146us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.426290 (+ 154us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.426332 (+ 42us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:56:22.434423 19920 raft_consensus.cc:1275] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 1 FOLLOWER]: Refusing update from remote peer 2c24cb05132a49aeb48e3a8c723e5857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:22.434419 19784 raft_consensus.cc:1275] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 1 FOLLOWER]: Refusing update from remote peer 2c24cb05132a49aeb48e3a8c723e5857: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:22.435420 19990 consensus_queue.cc:1048] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "dd9743c815ff4497978b86557a625bd3" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34849 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:22.435968 19990 consensus_queue.cc:1048] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [LEADER]: Connected to new peer: Peer: permanent_uuid: "664f9b2ee0804925b058cd06a1bd61b7" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 35221 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:22.451866 19999 mvcc.cc:204] Tried to move back new op lower bound from 7215419320041099264 to 7215419319901081600. Current Snapshot: MvccSnapshot[applied={T|T < 7215419320041099264}]
I20251027 14:56:22.459672 20000 mvcc.cc:204] Tried to move back new op lower bound from 7215419320041099264 to 7215419319901081600. Current Snapshot: MvccSnapshot[applied={T|T < 7215419320041099264}]
I20251027 14:56:22.468487 19501 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59492:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:22.468750 19501 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:59492:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:22.472242 19501 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 59350b9c4b134e74afc57be956c90624: Sending DeleteTablet for 3 replicas of tablet a0d7118b44cc42258c57ae5780c0516d
I20251027 14:56:22.473570 19900 tablet_service.cc:1552] Processing DeleteTablet for tablet a0d7118b44cc42258c57ae5780c0516d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:22 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:33920
I20251027 14:56:22.473583 19628 tablet_service.cc:1552] Processing DeleteTablet for tablet a0d7118b44cc42258c57ae5780c0516d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:22 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:42190
I20251027 14:56:22.474298 19764 tablet_service.cc:1552] Processing DeleteTablet for tablet a0d7118b44cc42258c57ae5780c0516d with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:22 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:49674
I20251027 14:56:22.475844 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19565
I20251027 14:56:22.483777 20009 tablet_replica.cc:333] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: stopping tablet replica
I20251027 14:56:22.484302 20009 raft_consensus.cc:2243] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:22.484884 20009 raft_consensus.cc:2272] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:22.489157 20009 ts_tablet_manager.cc:1916] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:22.492128 20007 tablet_replica.cc:333] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: stopping tablet replica
I20251027 14:56:22.492583 20007 raft_consensus.cc:2243] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:22.493088 20007 raft_consensus.cc:2272] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:22.496354 20008 tablet_replica.cc:333] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: stopping tablet replica
I20251027 14:56:22.496451 20009 ts_tablet_manager.cc:1929] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:22.496615 20009 log.cc:1199] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/a0d7118b44cc42258c57ae5780c0516d
I20251027 14:56:22.497112 20009 ts_tablet_manager.cc:1950] T a0d7118b44cc42258c57ae5780c0516d P 664f9b2ee0804925b058cd06a1bd61b7: Deleting consensus metadata
I20251027 14:56:22.497112 20008 raft_consensus.cc:2243] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:56:22.497982 20008 raft_consensus.cc:2272] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:22.499742 20007 ts_tablet_manager.cc:1916] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:22.499869 19486 catalog_manager.cc:4985] TS 664f9b2ee0804925b058cd06a1bd61b7 (127.0.104.2:35221): tablet a0d7118b44cc42258c57ae5780c0516d (table test-table [id=13d0d5f1d2714a2d8dddad425bce99ad]) successfully deleted
I20251027 14:56:22.499922 20008 ts_tablet_manager.cc:1916] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Deleting tablet data with delete state TABLET_DATA_DELETED
Oct 27 14:56:22 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:56:22.507069 20007 ts_tablet_manager.cc:1929] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:22.507233 20007 log.cc:1199] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/a0d7118b44cc42258c57ae5780c0516d
I20251027 14:56:22.507709 20007 ts_tablet_manager.cc:1950] T a0d7118b44cc42258c57ae5780c0516d P dd9743c815ff4497978b86557a625bd3: Deleting consensus metadata
I20251027 14:56:22.508071 20008 ts_tablet_manager.cc:1929] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:22.508297 20008 log.cc:1199] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_2.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/a0d7118b44cc42258c57ae5780c0516d
I20251027 14:56:22.508917 20008 ts_tablet_manager.cc:1950] T a0d7118b44cc42258c57ae5780c0516d P 2c24cb05132a49aeb48e3a8c723e5857: Deleting consensus metadata
I20251027 14:56:22.509747 19489 catalog_manager.cc:4985] TS dd9743c815ff4497978b86557a625bd3 (127.0.104.3:34849): tablet a0d7118b44cc42258c57ae5780c0516d (table test-table [id=13d0d5f1d2714a2d8dddad425bce99ad]) successfully deleted
I20251027 14:56:22.510856 19486 catalog_manager.cc:4985] TS 2c24cb05132a49aeb48e3a8c723e5857 (127.0.104.1:35483): tablet a0d7118b44cc42258c57ae5780c0516d (table test-table [id=13d0d5f1d2714a2d8dddad425bce99ad]) successfully deleted
I20251027 14:56:22.511966 19979 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.477188 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42206 (local address 127.0.104.1:35483)
1027 14:56:22.477385 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.477392 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.488966 (+ 11574us) server_negotiation.cc:408] Connection header received
1027 14:56:22.489479 (+ 513us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.489486 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.489570 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.489646 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:22.493584 (+ 3938us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.495043 (+ 1459us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.496832 (+ 1789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.497392 (+ 560us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.504464 (+ 7072us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:22.504531 (+ 67us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:22.504549 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:22.504601 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:22.508853 (+ 4252us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.509819 (+ 966us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.509829 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.509832 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.509933 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.510347 (+ 414us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.510351 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.510353 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.510716 (+ 363us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:22.511048 (+ 332us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.511461 (+ 413us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.511670 (+ 209us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:56:22.512634 19688 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:22.612752 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19565
I20251027 14:56:22.634183 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19701
Oct 27 14:56:22 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:56:22.649008 19983 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.634870 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49676 (local address 127.0.104.2:35221)
1027 14:56:22.635010 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.635014 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.635052 (+ 38us) server_negotiation.cc:408] Connection header received
1027 14:56:22.635270 (+ 218us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.635276 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.635394 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.635485 (+ 91us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:22.636920 (+ 1435us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.637835 (+ 915us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.639090 (+ 1255us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.639547 (+ 457us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.643130 (+ 3583us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:22.643181 (+ 51us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:22.643197 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:22.643239 (+ 42us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:22.646215 (+ 2976us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.647004 (+ 789us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.647012 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.647015 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.647104 (+ 89us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.647413 (+ 309us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.647417 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.647419 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.647737 (+ 318us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:22.648019 (+ 282us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.648519 (+ 500us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.648741 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":48}
I20251027 14:56:22.649722 19824 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:22.739341 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19701
I20251027 14:56:22.753867 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19837
Oct 27 14:56:22 dist-test-slave-hxxt krb5kdc[19454](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576979, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:56:22.769379 19982 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:22.754638 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33930 (local address 127.0.104.3:34849)
1027 14:56:22.754843 (+ 205us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:22.754849 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:22.754869 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:56:22.755046 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:22.755052 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:22.755118 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:22.755204 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:22.756681 (+ 1477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.757517 (+ 836us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:22.758724 (+ 1207us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:22.759151 (+ 427us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:22.763214 (+ 4063us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:22.763257 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:22.763274 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:22.763308 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:22.766444 (+ 3136us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.767322 (+ 878us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.767331 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.767334 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.767440 (+ 106us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:22.767795 (+ 355us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:22.767799 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:22.767802 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:22.768079 (+ 277us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:22.768371 (+ 292us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:22.768789 (+ 418us) server_negotiation.cc:300] Negotiation successful
1027 14:56:22.769034 (+ 245us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":86}
I20251027 14:56:22.770097 19960 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:22.859890 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19837
I20251027 14:56:22.873903 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 19470
I20251027 14:56:22.874995 19532 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:22.985276 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 19470
2025-10-27T14:56:23Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/2 (5168 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/3
2025-10-27T14:56:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:23Z Disabled control of system clock
I20251027 14:56:23.054375 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36533
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:36533
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251027 14:56:23.380735 20018 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:23.381063 20018 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:23.381191 20018 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:23.389532 20018 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:23.389699 20018 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:23.389763 20018 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:23.389832 20018 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:23.389909 20018 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:23.400018 20018 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:36533
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36533
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:23.401705 20018 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:23.403537 20018 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:23.411329 20023 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:23.411360 20024 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:23.411971 20026 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:23.412585 20018 server_base.cc:1047] running on GCE node
I20251027 14:56:23.413429 20018 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:23.414795 20018 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:23.416020 20018 hybrid_clock.cc:648] HybridClock initialized: now 1761576983415958 us; error 35 us; skew 500 ppm
I20251027 14:56:23.416340 20018 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:23.418449 20018 webserver.cc:492] Webserver started at http://127.0.104.62:35887/ using document root <none> and password file <none>
I20251027 14:56:23.419147 20018 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:23.419256 20018 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:23.419588 20018 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:23.422088 20018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c"
format_stamp: "Formatted at 2025-10-27 14:56:23 on dist-test-slave-hxxt"
server_key: "4355008ac660d1cbe905f6dc38f56052"
server_key_iv: "95253108051e21133c48e7132cabd9f6"
server_key_version: "encryptionkey@0"
I20251027 14:56:23.422830 20018 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c"
format_stamp: "Formatted at 2025-10-27 14:56:23 on dist-test-slave-hxxt"
server_key: "4355008ac660d1cbe905f6dc38f56052"
server_key_iv: "95253108051e21133c48e7132cabd9f6"
server_key_version: "encryptionkey@0"
I20251027 14:56:23.428009 20018 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:56:23.431394 20032 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:23.432689 20018 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:56:23.432842 20018 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c"
format_stamp: "Formatted at 2025-10-27 14:56:23 on dist-test-slave-hxxt"
server_key: "4355008ac660d1cbe905f6dc38f56052"
server_key_iv: "95253108051e21133c48e7132cabd9f6"
server_key_version: "encryptionkey@0"
I20251027 14:56:23.433029 20018 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:23.462180 20018 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:23.463043 20018 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:23.463294 20018 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:23.481338 20018 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:36533
I20251027 14:56:23.481357 20083 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:36533 every 8 connection(s)
I20251027 14:56:23.482762 20018 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:23.486095 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20018
I20251027 14:56:23.486274 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:23.486737 416 external_mini_cluster.cc:1468] Setting key 697f2aa0ec4afbe1c32fdcf612df4a78
I20251027 14:56:23.487283 20084 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:23.499540 20087 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:23.488679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.0.104.62:36533)
1027 14:56:23.489594 (+ 915us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:23.489605 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:23.489652 (+ 47us) server_negotiation.cc:408] Connection header received
1027 14:56:23.491351 (+ 1699us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:23.491374 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:23.491858 (+ 484us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:23.492249 (+ 391us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:23.493272 (+ 1023us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:23.494623 (+ 1351us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:23.496226 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:23.496757 (+ 531us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:23.497226 (+ 469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:23.497314 (+ 88us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:23.497328 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:23.497929 (+ 601us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:23.497981 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:23.498001 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:23.498086 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:23.498313 (+ 227us) server_negotiation.cc:300] Negotiation successful
1027 14:56:23.498612 (+ 299us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":353,"threads_started":1}
I20251027 14:56:23.500361 20084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c: Bootstrap starting.
I20251027 14:56:23.503400 20084 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:23.504410 20084 log.cc:826] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:23.507170 20084 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c: No bootstrap required, opened a new log
I20251027 14:56:23.512147 20084 raft_consensus.cc:359] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36533 } }
I20251027 14:56:23.512506 20084 raft_consensus.cc:385] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:23.512615 20084 raft_consensus.cc:740] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b60ba2438d7d4693aa2f7dcdb2492e2c, State: Initialized, Role: FOLLOWER
I20251027 14:56:23.513222 20084 consensus_queue.cc:260] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36533 } }
I20251027 14:56:23.513419 20084 raft_consensus.cc:399] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:23.513548 20084 raft_consensus.cc:493] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:23.513710 20084 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:23.515849 20084 raft_consensus.cc:515] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36533 } }
I20251027 14:56:23.516285 20084 leader_election.cc:304] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: b60ba2438d7d4693aa2f7dcdb2492e2c; no voters:
I20251027 14:56:23.516865 20084 leader_election.cc:290] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:23.517032 20089 raft_consensus.cc:2804] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:23.517611 20089 raft_consensus.cc:697] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [term 1 LEADER]: Becoming Leader. State: Replica: b60ba2438d7d4693aa2f7dcdb2492e2c, State: Running, Role: LEADER
I20251027 14:56:23.518074 20089 consensus_queue.cc:237] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36533 } }
I20251027 14:56:23.518729 20084 sys_catalog.cc:565] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:23.520759 20091 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [sys.catalog]: SysCatalogTable state changed. Reason: New leader b60ba2438d7d4693aa2f7dcdb2492e2c. Latest consensus state: current_term: 1 leader_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36533 } } }
I20251027 14:56:23.521082 20091 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:23.521081 20090 sys_catalog.cc:455] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "b60ba2438d7d4693aa2f7dcdb2492e2c" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36533 } } }
I20251027 14:56:23.521401 20090 sys_catalog.cc:458] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:23.523105 20094 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:23.529027 20094 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:23.538513 20094 catalog_manager.cc:1357] Generated new cluster ID: bc19e238b3ba40d3ba6aa0acbca6e111
I20251027 14:56:23.538645 20094 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:23.551266 20094 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:23.553053 20094 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:23.572324 20094 catalog_manager.cc:6022] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c: Generated new TSK 0
I20251027 14:56:23.573417 20094 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:56:23.583740 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36533
--builtin_ntp_servers=127.0.104.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251027 14:56:23.907788 20108 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:23.908125 20108 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:23.908248 20108 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:23.916286 20108 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:23.916406 20108 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:23.916569 20108 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:56:23.926249 20108 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:23.927851 20108 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:23.929680 20108 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:23.938577 20113 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:23.938616 20114 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:23.939252 20116 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:23.939620 20108 server_base.cc:1047] running on GCE node
I20251027 14:56:23.940160 20108 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:23.941054 20108 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:23.942260 20108 hybrid_clock.cc:648] HybridClock initialized: now 1761576983942198 us; error 37 us; skew 500 ppm
I20251027 14:56:23.942519 20108 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:23.944535 20108 webserver.cc:492] Webserver started at http://127.0.104.1:37903/ using document root <none> and password file <none>
I20251027 14:56:23.945201 20108 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:23.945314 20108 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:23.945636 20108 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:23.948163 20108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "3e72ef987c1f41dca1ad04931e854506"
format_stamp: "Formatted at 2025-10-27 14:56:23 on dist-test-slave-hxxt"
server_key: "b96c71d91227432709c19d8da5e390c8"
server_key_iv: "2b17c96a0cd35c0966323f1e2e45a437"
server_key_version: "encryptionkey@0"
I20251027 14:56:23.948933 20108 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "3e72ef987c1f41dca1ad04931e854506"
format_stamp: "Formatted at 2025-10-27 14:56:23 on dist-test-slave-hxxt"
server_key: "b96c71d91227432709c19d8da5e390c8"
server_key_iv: "2b17c96a0cd35c0966323f1e2e45a437"
server_key_version: "encryptionkey@0"
I20251027 14:56:23.954021 20108 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:56:23.957355 20122 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:23.958598 20108 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:56:23.958745 20108 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "3e72ef987c1f41dca1ad04931e854506"
format_stamp: "Formatted at 2025-10-27 14:56:23 on dist-test-slave-hxxt"
server_key: "b96c71d91227432709c19d8da5e390c8"
server_key_iv: "2b17c96a0cd35c0966323f1e2e45a437"
server_key_version: "encryptionkey@0"
I20251027 14:56:23.958936 20108 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:23.977254 20108 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:23.978093 20108 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:23.978356 20108 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:23.979251 20108 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:23.980867 20108 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:23.980952 20108 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:23.981079 20108 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:23.981133 20108 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.011317 20108 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:37893
I20251027 14:56:24.011348 20234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:37893 every 8 connection(s)
I20251027 14:56:24.012777 20108 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:56:24.016526 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20108
I20251027 14:56:24.016711 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:56:24.017156 416 external_mini_cluster.cc:1468] Setting key 93465bf3380d690d23ebb7a78fc9bae2
I20251027 14:56:24.019740 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36533
--builtin_ntp_servers=127.0.104.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251027 14:56:24.024309 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.015374 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:57143 (local address 127.0.104.62:36533)
1027 14:56:24.015787 (+ 413us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.015792 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.017130 (+ 1338us) server_negotiation.cc:408] Connection header received
1027 14:56:24.018743 (+ 1613us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.018750 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.018832 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.018930 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:24.020327 (+ 1397us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.021231 (+ 904us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.022287 (+ 1056us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.022802 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.023467 (+ 665us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:24.023506 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:24.023511 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:24.023530 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:24.023573 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:24.023588 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:24.023672 (+ 84us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.023895 (+ 223us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.024009 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":315,"thread_start_us":216,"threads_started":1}
I20251027 14:56:24.025152 20238 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.015923 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36533 (local address 127.0.104.1:57143)
1027 14:56:24.016970 (+ 1047us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:24.017007 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:24.018474 (+ 1467us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:24.019208 (+ 734us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:24.019218 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:24.019312 (+ 94us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:24.020171 (+ 859us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.020190 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.021371 (+ 1181us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.021375 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:24.022160 (+ 785us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.022170 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.022412 (+ 242us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.023186 (+ 774us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:24.023189 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:24.023222 (+ 33us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:24.023296 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:24.023672 (+ 376us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:24.023679 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:24.023685 (+ 6us) client_negotiation.cc:769] Sending connection context
1027 14:56:24.023884 (+ 199us) client_negotiation.cc:240] Negotiation successful
1027 14:56:24.024174 (+ 290us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":539,"thread_start_us":333,"threads_started":1}
I20251027 14:56:24.026753 20235 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36533
I20251027 14:56:24.027084 20235 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:24.027843 20235 heartbeater.cc:507] Master 127.0.104.62:36533 requested a full tablet report, sending...
I20251027 14:56:24.030131 20049 ts_manager.cc:194] Registered new tserver with Master: 3e72ef987c1f41dca1ad04931e854506 (127.0.104.1:37893)
I20251027 14:56:24.032164 20049 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:57143
W20251027 14:56:24.340188 20239 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:24.340479 20239 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:24.340569 20239 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:24.348488 20239 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:24.348574 20239 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:24.348698 20239 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:56:24.358136 20239 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:24.359678 20239 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:24.361426 20239 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:24.370525 20245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:24.370473 20244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:24.370926 20247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:24.371716 20239 server_base.cc:1047] running on GCE node
I20251027 14:56:24.372239 20239 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:24.373095 20239 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:24.374306 20239 hybrid_clock.cc:648] HybridClock initialized: now 1761576984374241 us; error 39 us; skew 500 ppm
I20251027 14:56:24.374576 20239 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:24.376597 20239 webserver.cc:492] Webserver started at http://127.0.104.2:36701/ using document root <none> and password file <none>
I20251027 14:56:24.377298 20239 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:24.377400 20239 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:24.377717 20239 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:24.380249 20239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "e5e59be94be64aa5807d93aaefb3efab"
format_stamp: "Formatted at 2025-10-27 14:56:24 on dist-test-slave-hxxt"
server_key: "abf2a7e68265a30cd39adb5278ef3430"
server_key_iv: "c08634736034f0949507f89d1e0e8e7e"
server_key_version: "encryptionkey@0"
I20251027 14:56:24.381197 20239 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "e5e59be94be64aa5807d93aaefb3efab"
format_stamp: "Formatted at 2025-10-27 14:56:24 on dist-test-slave-hxxt"
server_key: "abf2a7e68265a30cd39adb5278ef3430"
server_key_iv: "c08634736034f0949507f89d1e0e8e7e"
server_key_version: "encryptionkey@0"
I20251027 14:56:24.386312 20239 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:56:24.389684 20253 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.390942 20239 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:56:24.391096 20239 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "e5e59be94be64aa5807d93aaefb3efab"
format_stamp: "Formatted at 2025-10-27 14:56:24 on dist-test-slave-hxxt"
server_key: "abf2a7e68265a30cd39adb5278ef3430"
server_key_iv: "c08634736034f0949507f89d1e0e8e7e"
server_key_version: "encryptionkey@0"
I20251027 14:56:24.391285 20239 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:24.422257 20239 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:24.423086 20239 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:24.423347 20239 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:24.424295 20239 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:24.426003 20239 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:24.426084 20239 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.426192 20239 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:24.426240 20239 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.456219 20239 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:40583
I20251027 14:56:24.456231 20365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:40583 every 8 connection(s)
I20251027 14:56:24.457674 20239 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:56:24.465646 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20239
I20251027 14:56:24.465837 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:56:24.466325 416 external_mini_cluster.cc:1468] Setting key 81d88dcca84f8926f9b0f17852c51e1a
I20251027 14:56:24.469272 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36533
--builtin_ntp_servers=127.0.104.20:44483
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251027 14:56:24.469460 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.460305 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:34319 (local address 127.0.104.62:36533)
1027 14:56:24.460481 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.460485 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.461889 (+ 1404us) server_negotiation.cc:408] Connection header received
1027 14:56:24.463539 (+ 1650us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.463547 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.463630 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.463718 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:24.464993 (+ 1275us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.466168 (+ 1175us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.467473 (+ 1305us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.467932 (+ 459us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.468621 (+ 689us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:24.468656 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:24.468659 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:24.468674 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:24.468704 (+ 30us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:24.468716 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:24.468787 (+ 71us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.469069 (+ 282us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.469198 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:56:24.470427 20368 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.460846 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36533 (local address 127.0.104.2:34319)
1027 14:56:24.461729 (+ 883us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:24.461764 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:24.463172 (+ 1408us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:24.463970 (+ 798us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:24.463978 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:24.464069 (+ 91us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:24.464820 (+ 751us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.464838 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.466338 (+ 1500us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.466343 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:24.467332 (+ 989us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.467348 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.467568 (+ 220us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.468310 (+ 742us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:24.468314 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:24.468351 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:24.468454 (+ 103us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:24.468802 (+ 348us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:24.468812 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:24.468820 (+ 8us) client_negotiation.cc:769] Sending connection context
1027 14:56:24.469032 (+ 212us) client_negotiation.cc:240] Negotiation successful
1027 14:56:24.469397 (+ 365us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":470,"thread_start_us":295,"threads_started":1}
I20251027 14:56:24.471771 20366 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36533
I20251027 14:56:24.472102 20366 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:24.472908 20366 heartbeater.cc:507] Master 127.0.104.62:36533 requested a full tablet report, sending...
I20251027 14:56:24.474287 20049 ts_manager.cc:194] Registered new tserver with Master: e5e59be94be64aa5807d93aaefb3efab (127.0.104.2:40583)
I20251027 14:56:24.475273 20049 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:34319
W20251027 14:56:24.785032 20369 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:24.785305 20369 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:24.785393 20369 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:24.793323 20369 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:24.793409 20369 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:24.793534 20369 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:56:24.803035 20369 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:44483
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:36533
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:24.804597 20369 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:24.806327 20369 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:24.814927 20374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:24.814901 20375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:24.815968 20377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:24.816167 20369 server_base.cc:1047] running on GCE node
I20251027 14:56:24.816793 20369 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:24.817668 20369 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:24.818872 20369 hybrid_clock.cc:648] HybridClock initialized: now 1761576984818813 us; error 36 us; skew 500 ppm
I20251027 14:56:24.819134 20369 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:24.821218 20369 webserver.cc:492] Webserver started at http://127.0.104.3:34055/ using document root <none> and password file <none>
I20251027 14:56:24.821910 20369 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:24.822023 20369 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:24.822361 20369 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:24.824997 20369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "0d20c208f08147a69ee9fd89f2397fcb"
format_stamp: "Formatted at 2025-10-27 14:56:24 on dist-test-slave-hxxt"
server_key: "c17c321ff4a74fdf62f390f05cff1ed4"
server_key_iv: "668d06efd203dd5b03a79746cf8f48ed"
server_key_version: "encryptionkey@0"
I20251027 14:56:24.825726 20369 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "0d20c208f08147a69ee9fd89f2397fcb"
format_stamp: "Formatted at 2025-10-27 14:56:24 on dist-test-slave-hxxt"
server_key: "c17c321ff4a74fdf62f390f05cff1ed4"
server_key_iv: "668d06efd203dd5b03a79746cf8f48ed"
server_key_version: "encryptionkey@0"
I20251027 14:56:24.830919 20369 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:56:24.834424 20383 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.835633 20369 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:56:24.835783 20369 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "0d20c208f08147a69ee9fd89f2397fcb"
format_stamp: "Formatted at 2025-10-27 14:56:24 on dist-test-slave-hxxt"
server_key: "c17c321ff4a74fdf62f390f05cff1ed4"
server_key_iv: "668d06efd203dd5b03a79746cf8f48ed"
server_key_version: "encryptionkey@0"
I20251027 14:56:24.835954 20369 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:24.867856 20369 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:24.868824 20369 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:24.869104 20369 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:24.870069 20369 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:24.871713 20369 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:24.871814 20369 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.871925 20369 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:24.871969 20369 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:24.902482 20369 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:43259
I20251027 14:56:24.902519 20495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:43259 every 8 connection(s)
I20251027 14:56:24.903810 20369 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:56:24.912801 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20369
I20251027 14:56:24.912995 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:56:24.913479 416 external_mini_cluster.cc:1468] Setting key eb561835de8d65f548d9bada76d534fe
I20251027 14:56:24.915604 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.906400 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:40085 (local address 127.0.104.62:36533)
1027 14:56:24.906568 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.906573 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.908072 (+ 1499us) server_negotiation.cc:408] Connection header received
1027 14:56:24.909709 (+ 1637us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.909715 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.909805 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.909895 (+ 90us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:24.911178 (+ 1283us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.912035 (+ 857us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.913453 (+ 1418us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.913819 (+ 366us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.914654 (+ 835us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:24.914689 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:24.914694 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:24.914713 (+ 19us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:24.914752 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:24.914768 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:24.914854 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.915181 (+ 327us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.915322 (+ 141us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:56:24.916667 20498 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.906938 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36533 (local address 127.0.104.3:40085)
1027 14:56:24.907907 (+ 969us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:24.907944 (+ 37us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:24.909426 (+ 1482us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:24.910151 (+ 725us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:24.910166 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:24.910259 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:24.910998 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.911016 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.912185 (+ 1169us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.912189 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:24.913313 (+ 1124us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.913328 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.913593 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.914388 (+ 795us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:24.914392 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:24.914429 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:24.914502 (+ 73us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:24.914879 (+ 377us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:24.914889 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:24.914897 (+ 8us) client_negotiation.cc:769] Sending connection context
1027 14:56:24.915135 (+ 238us) client_negotiation.cc:240] Negotiation successful
1027 14:56:24.915486 (+ 351us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":526,"thread_start_us":312,"threads_started":1}
I20251027 14:56:24.917925 20496 heartbeater.cc:344] Connected to a master server at 127.0.104.62:36533
I20251027 14:56:24.918243 20496 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:24.918980 20496 heartbeater.cc:507] Master 127.0.104.62:36533 requested a full tablet report, sending...
I20251027 14:56:24.920353 20049 ts_manager.cc:194] Registered new tserver with Master: 0d20c208f08147a69ee9fd89f2397fcb (127.0.104.3:43259)
I20251027 14:56:24.921159 20049 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:40085
I20251027 14:56:24.929075 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:56:24.938330 20237 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.933233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53858 (local address 127.0.104.62:36533)
1027 14:56:24.933443 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.933450 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.933681 (+ 231us) server_negotiation.cc:408] Connection header received
1027 14:56:24.933810 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.933814 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.933881 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.933959 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:24.934644 (+ 685us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.935504 (+ 860us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.936841 (+ 1337us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.937216 (+ 375us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.937685 (+ 469us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:24.937710 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:24.937714 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:24.937729 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:24.937755 (+ 26us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:24.937766 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:24.937864 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.937989 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.938030 (+ 41us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":74}
I20251027 14:56:24.941776 20049 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:53858:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:56:24.946321 20049 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:56:24.980702 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.959597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39770 (local address 127.0.104.2:40583)
1027 14:56:24.971081 (+ 11484us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.971093 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.972744 (+ 1651us) server_negotiation.cc:408] Connection header received
1027 14:56:24.973032 (+ 288us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.973043 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.973333 (+ 290us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.973540 (+ 207us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:24.974951 (+ 1411us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.976573 (+ 1622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.978763 (+ 2190us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.979850 (+ 1087us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.980077 (+ 227us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.980212 (+ 135us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.980426 (+ 214us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11269,"thread_start_us":11528,"threads_started":1}
I20251027 14:56:24.981184 20510 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.958630 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34348)
1027 14:56:24.971825 (+ 13195us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:24.971872 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:24.972082 (+ 210us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:24.974404 (+ 2322us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:24.974412 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:24.974441 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:24.975166 (+ 725us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.975179 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.978833 (+ 3654us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.978839 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:24.980544 (+ 1705us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.980559 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.980688 (+ 129us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.980712 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:56:24.980768 (+ 56us) client_negotiation.cc:240] Negotiation successful
1027 14:56:24.980886 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12962,"thread_start_us":13782,"threads_started":1}
I20251027 14:56:24.979562 20508 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.958969 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:40583 (local address 127.0.0.1:39770)
1027 14:56:24.972624 (+ 13655us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:24.972642 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:24.972834 (+ 192us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:24.973864 (+ 1030us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:24.973879 (+ 15us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:24.973911 (+ 32us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:24.974754 (+ 843us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.974771 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.976785 (+ 2014us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.976790 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:24.978590 (+ 1800us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.978611 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.978818 (+ 207us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.978884 (+ 66us) client_negotiation.cc:769] Sending connection context
1027 14:56:24.979030 (+ 146us) client_negotiation.cc:240] Negotiation successful
1027 14:56:24.979215 (+ 185us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13559,"thread_start_us":12492,"threads_started":1}
I20251027 14:56:24.982555 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.958912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34348 (local address 127.0.104.1:37893)
1027 14:56:24.973786 (+ 14874us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.973804 (+ 18us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.973834 (+ 30us) server_negotiation.cc:408] Connection header received
1027 14:56:24.973921 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.973928 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.974163 (+ 235us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.976625 (+ 2462us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:24.976904 (+ 279us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.978644 (+ 1740us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.980939 (+ 2295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.981993 (+ 1054us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.982131 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.982225 (+ 94us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.982364 (+ 139us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14689,"thread_start_us":14579,"threads_started":1}
I20251027 14:56:24.983537 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.973940 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:43259 (local address 127.0.0.1:42062)
1027 14:56:24.977517 (+ 3577us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:24.977531 (+ 14us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:24.977647 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:24.978144 (+ 497us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:24.978149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:24.978178 (+ 29us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:24.978841 (+ 663us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.978855 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.981279 (+ 2424us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.981284 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:24.982888 (+ 1604us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:24.982904 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.983052 (+ 148us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.983074 (+ 22us) client_negotiation.cc:769] Sending connection context
1027 14:56:24.983148 (+ 74us) client_negotiation.cc:240] Negotiation successful
1027 14:56:24.983257 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":3487,"thread_start_us":3319,"threads_started":1}
I20251027 14:56:24.984894 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:24.973176 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42062 (local address 127.0.104.3:43259)
1027 14:56:24.977212 (+ 4036us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:24.977222 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:24.977643 (+ 421us) server_negotiation.cc:408] Connection header received
1027 14:56:24.977793 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:24.977799 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:24.977965 (+ 166us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:24.978155 (+ 190us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:24.979024 (+ 869us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.981125 (+ 2101us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:24.983048 (+ 1923us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:24.984153 (+ 1105us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:24.984329 (+ 176us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:24.984472 (+ 143us) server_negotiation.cc:300] Negotiation successful
1027 14:56:24.984642 (+ 170us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":3897,"thread_start_us":3738,"threads_started":1}
I20251027 14:56:24.986342 20301 tablet_service.cc:1505] Processing CreateTablet for tablet 98bf3b538315451b82d476d41cfee528 (DEFAULT_TABLE table=test-table [id=e40648ebe3e74814a6896e28cd69bf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:24.986734 20170 tablet_service.cc:1505] Processing CreateTablet for tablet 98bf3b538315451b82d476d41cfee528 (DEFAULT_TABLE table=test-table [id=e40648ebe3e74814a6896e28cd69bf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:24.987564 20301 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bf3b538315451b82d476d41cfee528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:24.988127 20170 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bf3b538315451b82d476d41cfee528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:24.989080 20431 tablet_service.cc:1505] Processing CreateTablet for tablet 98bf3b538315451b82d476d41cfee528 (DEFAULT_TABLE table=test-table [id=e40648ebe3e74814a6896e28cd69bf62]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:24.990444 20431 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 98bf3b538315451b82d476d41cfee528. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:24.998668 20514 tablet_bootstrap.cc:492] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Bootstrap starting.
I20251027 14:56:25.001379 20515 tablet_bootstrap.cc:492] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Bootstrap starting.
I20251027 14:56:25.002071 20514 tablet_bootstrap.cc:654] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:25.003291 20516 tablet_bootstrap.cc:492] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Bootstrap starting.
I20251027 14:56:25.003882 20514 log.cc:826] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:25.005528 20515 tablet_bootstrap.cc:654] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:25.006729 20516 tablet_bootstrap.cc:654] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:25.006803 20514 tablet_bootstrap.cc:492] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: No bootstrap required, opened a new log
I20251027 14:56:25.006858 20515 log.cc:826] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:25.007164 20514 ts_tablet_manager.cc:1403] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.001s
I20251027 14:56:25.007925 20516 log.cc:826] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:25.010311 20515 tablet_bootstrap.cc:492] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: No bootstrap required, opened a new log
I20251027 14:56:25.010376 20516 tablet_bootstrap.cc:492] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: No bootstrap required, opened a new log
I20251027 14:56:25.010610 20515 ts_tablet_manager.cc:1403] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Time spent bootstrapping tablet: real 0.010s user 0.007s sys 0.000s
I20251027 14:56:25.010633 20516 ts_tablet_manager.cc:1403] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Time spent bootstrapping tablet: real 0.008s user 0.005s sys 0.000s
I20251027 14:56:25.014358 20514 raft_consensus.cc:359] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.014775 20514 raft_consensus.cc:385] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:25.014881 20514 raft_consensus.cc:740] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e5e59be94be64aa5807d93aaefb3efab, State: Initialized, Role: FOLLOWER
I20251027 14:56:25.015691 20514 consensus_queue.cc:260] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.017696 20516 raft_consensus.cc:359] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.018043 20516 raft_consensus.cc:385] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:25.018121 20516 raft_consensus.cc:740] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d20c208f08147a69ee9fd89f2397fcb, State: Initialized, Role: FOLLOWER
I20251027 14:56:25.018818 20516 consensus_queue.cc:260] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.021561 20515 raft_consensus.cc:359] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.021977 20515 raft_consensus.cc:385] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:25.022091 20515 raft_consensus.cc:740] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3e72ef987c1f41dca1ad04931e854506, State: Initialized, Role: FOLLOWER
I20251027 14:56:25.022801 20515 consensus_queue.cc:260] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.023800 20516 ts_tablet_manager.cc:1434] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Time spent starting tablet: real 0.013s user 0.013s sys 0.000s
I20251027 14:56:25.024672 20496 heartbeater.cc:499] Master 127.0.104.62:36533 was elected leader, sending a full tablet report...
I20251027 14:56:25.025290 20514 ts_tablet_manager.cc:1434] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Time spent starting tablet: real 0.018s user 0.007s sys 0.012s
I20251027 14:56:25.026293 20366 heartbeater.cc:499] Master 127.0.104.62:36533 was elected leader, sending a full tablet report...
I20251027 14:56:25.027855 20235 heartbeater.cc:499] Master 127.0.104.62:36533 was elected leader, sending a full tablet report...
I20251027 14:56:25.029148 20515 ts_tablet_manager.cc:1434] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Time spent starting tablet: real 0.018s user 0.014s sys 0.000s
I20251027 14:56:25.062732 20520 raft_consensus.cc:493] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:25.063005 20520 raft_consensus.cc:515] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.064886 20520 leader_election.cc:290] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 3e72ef987c1f41dca1ad04931e854506 (127.0.104.1:37893), 0d20c208f08147a69ee9fd89f2397fcb (127.0.104.3:43259)
I20251027 14:56:25.071002 20523 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.065333 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.104.2:55385)
1027 14:56:25.065896 (+ 563us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.065912 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:25.066051 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:25.066512 (+ 461us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:25.066518 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:25.066546 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:25.067252 (+ 706us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:25.067265 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.068897 (+ 1632us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.068904 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:25.070388 (+ 1484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:25.070402 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.070544 (+ 142us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.070566 (+ 22us) client_negotiation.cc:769] Sending connection context
1027 14:56:25.070618 (+ 52us) client_negotiation.cc:240] Negotiation successful
1027 14:56:25.070726 (+ 108us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":476,"thread_start_us":322,"threads_started":1}
I20251027 14:56:25.071631 20524 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.066029 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:43259 (local address 127.0.104.2:56149)
1027 14:56:25.066563 (+ 534us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.066579 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:25.066715 (+ 136us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:25.067012 (+ 297us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:25.067017 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:25.067041 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:25.067629 (+ 588us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:25.067638 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.069320 (+ 1682us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.069326 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:25.070733 (+ 1407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:25.070748 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.070922 (+ 174us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.070948 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:56:25.071020 (+ 72us) client_negotiation.cc:240] Negotiation successful
1027 14:56:25.071153 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":447,"thread_start_us":253,"threads_started":1}
I20251027 14:56:25.072093 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.065298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:55385 (local address 127.0.104.1:37893)
1027 14:56:25.065450 (+ 152us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:25.065455 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:25.066000 (+ 545us) server_negotiation.cc:408] Connection header received
1027 14:56:25.066249 (+ 249us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:25.066257 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:25.066371 (+ 114us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:25.066481 (+ 110us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:25.067427 (+ 946us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.068681 (+ 1254us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.070558 (+ 1877us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.071644 (+ 1086us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.071694 (+ 50us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:25.071777 (+ 83us) server_negotiation.cc:300] Negotiation successful
1027 14:56:25.071910 (+ 133us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:56:25.072278 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.066240 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:56149 (local address 127.0.104.3:43259)
1027 14:56:25.066415 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:25.066421 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:25.066673 (+ 252us) server_negotiation.cc:408] Connection header received
1027 14:56:25.066828 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:25.066832 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:25.066892 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:25.066970 (+ 78us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:25.067762 (+ 792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.069061 (+ 1299us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.070904 (+ 1843us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.071860 (+ 956us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.071903 (+ 43us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:25.071968 (+ 65us) server_negotiation.cc:300] Negotiation successful
1027 14:56:25.072061 (+ 93us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251027 14:56:25.073179 20451 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98bf3b538315451b82d476d41cfee528" candidate_uuid: "e5e59be94be64aa5807d93aaefb3efab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d20c208f08147a69ee9fd89f2397fcb" is_pre_election: true
I20251027 14:56:25.073179 20190 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98bf3b538315451b82d476d41cfee528" candidate_uuid: "e5e59be94be64aa5807d93aaefb3efab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e72ef987c1f41dca1ad04931e854506" is_pre_election: true
I20251027 14:56:25.073716 20451 raft_consensus.cc:2468] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e59be94be64aa5807d93aaefb3efab in term 0.
I20251027 14:56:25.073716 20190 raft_consensus.cc:2468] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e5e59be94be64aa5807d93aaefb3efab in term 0.
I20251027 14:56:25.074388 20257 leader_election.cc:304] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 3e72ef987c1f41dca1ad04931e854506, e5e59be94be64aa5807d93aaefb3efab; no voters:
I20251027 14:56:25.074736 20520 raft_consensus.cc:2804] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:56:25.074869 20520 raft_consensus.cc:493] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:56:25.074970 20520 raft_consensus.cc:3060] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:25.077306 20520 raft_consensus.cc:515] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.077853 20520 leader_election.cc:290] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [CANDIDATE]: Term 1 election: Requested vote from peers 3e72ef987c1f41dca1ad04931e854506 (127.0.104.1:37893), 0d20c208f08147a69ee9fd89f2397fcb (127.0.104.3:43259)
I20251027 14:56:25.078292 20190 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98bf3b538315451b82d476d41cfee528" candidate_uuid: "e5e59be94be64aa5807d93aaefb3efab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "3e72ef987c1f41dca1ad04931e854506"
I20251027 14:56:25.078366 20451 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "98bf3b538315451b82d476d41cfee528" candidate_uuid: "e5e59be94be64aa5807d93aaefb3efab" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d20c208f08147a69ee9fd89f2397fcb"
I20251027 14:56:25.078508 20190 raft_consensus.cc:3060] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:25.078532 20451 raft_consensus.cc:3060] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:25.081924 20451 raft_consensus.cc:2468] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e59be94be64aa5807d93aaefb3efab in term 1.
I20251027 14:56:25.081924 20190 raft_consensus.cc:2468] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e5e59be94be64aa5807d93aaefb3efab in term 1.
I20251027 14:56:25.082326 20257 leader_election.cc:304] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d20c208f08147a69ee9fd89f2397fcb, e5e59be94be64aa5807d93aaefb3efab; no voters:
I20251027 14:56:25.082580 20520 raft_consensus.cc:2804] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:25.082993 20520 raft_consensus.cc:697] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 1 LEADER]: Becoming Leader. State: Replica: e5e59be94be64aa5807d93aaefb3efab, State: Running, Role: LEADER
I20251027 14:56:25.083498 20520 consensus_queue.cc:237] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } }
I20251027 14:56:25.088218 20049 catalog_manager.cc:5649] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab reported cstate change: term changed from 0 to 1, leader changed from <none> to e5e59be94be64aa5807d93aaefb3efab (127.0.104.2). New cstate: current_term: 1 leader_uuid: "e5e59be94be64aa5807d93aaefb3efab" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "e5e59be94be64aa5807d93aaefb3efab" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 40583 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 } health_report { overall_health: UNKNOWN } } }
I20251027 14:56:25.109761 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.104235 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39782 (local address 127.0.104.2:40583)
1027 14:56:25.104447 (+ 212us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:25.104453 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:25.104529 (+ 76us) server_negotiation.cc:408] Connection header received
1027 14:56:25.104692 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:25.104697 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:25.104771 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:25.104888 (+ 117us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:56:25.105415 (+ 527us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.106333 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.107794 (+ 1461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.108178 (+ 384us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.108584 (+ 406us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:56:25.109095 (+ 511us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:56:25.109233 (+ 138us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:25.109455 (+ 222us) server_negotiation.cc:300] Negotiation successful
1027 14:56:25.109503 (+ 48us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:56:25.117936 20451 raft_consensus.cc:1275] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 1 FOLLOWER]: Refusing update from remote peer e5e59be94be64aa5807d93aaefb3efab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:25.117936 20190 raft_consensus.cc:1275] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 1 FOLLOWER]: Refusing update from remote peer e5e59be94be64aa5807d93aaefb3efab: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:25.118947 20525 consensus_queue.cc:1048] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [LEADER]: Connected to new peer: Peer: permanent_uuid: "3e72ef987c1f41dca1ad04931e854506" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 37893 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:25.119246 20520 consensus_queue.cc:1048] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [LEADER]: Connected to new peer: Peer: permanent_uuid: "0d20c208f08147a69ee9fd89f2397fcb" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 43259 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:25.130028 20528 mvcc.cc:204] Tried to move back new op lower bound from 7215419331031961600 to 7215419330906353664. Current Snapshot: MvccSnapshot[applied={T|T < 7215419331031961600}]
I20251027 14:56:25.143340 20049 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:53858:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:25.143574 20049 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:53858:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:25.147109 20049 catalog_manager.cc:5936] T 00000000000000000000000000000000 P b60ba2438d7d4693aa2f7dcdb2492e2c: Sending DeleteTablet for 3 replicas of tablet 98bf3b538315451b82d476d41cfee528
I20251027 14:56:25.148298 20301 tablet_service.cc:1552] Processing DeleteTablet for tablet 98bf3b538315451b82d476d41cfee528 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:25 UTC) from {username='slave'} at 127.0.0.1:39770
I20251027 14:56:25.148335 20170 tablet_service.cc:1552] Processing DeleteTablet for tablet 98bf3b538315451b82d476d41cfee528 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:25 UTC) from {username='slave'} at 127.0.0.1:34348
I20251027 14:56:25.149261 20431 tablet_service.cc:1552] Processing DeleteTablet for tablet 98bf3b538315451b82d476d41cfee528 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:25 UTC) from {username='slave'} at 127.0.0.1:42062
I20251027 14:56:25.149920 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20108
I20251027 14:56:25.156936 20538 tablet_replica.cc:333] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: stopping tablet replica
I20251027 14:56:25.157414 20538 raft_consensus.cc:2243] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:56:25.157672 20540 tablet_replica.cc:333] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: stopping tablet replica
I20251027 14:56:25.158054 20540 raft_consensus.cc:2243] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:25.158087 20538 raft_consensus.cc:2272] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:25.158555 20540 raft_consensus.cc:2272] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:25.158870 20539 tablet_replica.cc:333] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: stopping tablet replica
I20251027 14:56:25.159271 20539 raft_consensus.cc:2243] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:25.159767 20539 raft_consensus.cc:2272] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:25.161057 20540 ts_tablet_manager.cc:1916] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:25.161743 20538 ts_tablet_manager.cc:1916] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:25.166131 20539 ts_tablet_manager.cc:1916] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:25.168603 20540 ts_tablet_manager.cc:1929] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:25.168792 20540 log.cc:1199] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/98bf3b538315451b82d476d41cfee528
I20251027 14:56:25.169242 20538 ts_tablet_manager.cc:1929] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:25.169416 20538 log.cc:1199] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/98bf3b538315451b82d476d41cfee528
I20251027 14:56:25.169852 20538 ts_tablet_manager.cc:1950] T 98bf3b538315451b82d476d41cfee528 P e5e59be94be64aa5807d93aaefb3efab: Deleting consensus metadata
I20251027 14:56:25.169413 20540 ts_tablet_manager.cc:1950] T 98bf3b538315451b82d476d41cfee528 P 0d20c208f08147a69ee9fd89f2397fcb: Deleting consensus metadata
I20251027 14:56:25.171638 20035 catalog_manager.cc:4985] TS e5e59be94be64aa5807d93aaefb3efab (127.0.104.2:40583): tablet 98bf3b538315451b82d476d41cfee528 (table test-table [id=e40648ebe3e74814a6896e28cd69bf62]) successfully deleted
I20251027 14:56:25.171975 20036 catalog_manager.cc:4985] TS 0d20c208f08147a69ee9fd89f2397fcb (127.0.104.3:43259): tablet 98bf3b538315451b82d476d41cfee528 (table test-table [id=e40648ebe3e74814a6896e28cd69bf62]) successfully deleted
I20251027 14:56:25.172809 20509 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.151251 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34360 (local address 127.0.104.1:37893)
1027 14:56:25.151426 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:25.151433 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:25.157009 (+ 5576us) server_negotiation.cc:408] Connection header received
1027 14:56:25.157215 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:25.157223 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:25.157333 (+ 110us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:25.157426 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:25.158470 (+ 1044us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.159963 (+ 1493us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.161934 (+ 1971us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.162398 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.163700 (+ 1302us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:25.163736 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:25.163753 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:25.164500 (+ 747us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:25.164556 (+ 56us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:25.164580 (+ 24us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:25.164669 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:25.172334 (+ 7665us) server_negotiation.cc:300] Negotiation successful
1027 14:56:25.172491 (+ 157us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:56:25.173486 20230 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:25.173470 20539 ts_tablet_manager.cc:1929] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:25.173717 20539 log.cc:1199] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_3.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/98bf3b538315451b82d476d41cfee528
I20251027 14:56:25.262058 20539 ts_tablet_manager.cc:1950] T 98bf3b538315451b82d476d41cfee528 P 3e72ef987c1f41dca1ad04931e854506: Deleting consensus metadata
I20251027 14:56:25.262178 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20108
I20251027 14:56:25.283882 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20239
W20251027 14:56:25.283882 20036 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:37893 (error 108)
I20251027 14:56:25.285892 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.285415 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34366)
1027 14:56:25.285601 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.285717 (+ 116us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":63}
W20251027 14:56:25.286159 20036 catalog_manager.cc:4712] TS 3e72ef987c1f41dca1ad04931e854506 (127.0.104.1:37893): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 98bf3b538315451b82d476d41cfee528: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
I20251027 14:56:25.291337 20511 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.285193 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39786 (local address 127.0.104.2:40583)
1027 14:56:25.285389 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:25.285395 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:25.285616 (+ 221us) server_negotiation.cc:408] Connection header received
1027 14:56:25.285806 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:25.285810 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:25.285936 (+ 126us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:25.286023 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:25.286837 (+ 814us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.287823 (+ 986us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.289149 (+ 1326us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.289638 (+ 489us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.290018 (+ 380us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:25.290049 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:25.290064 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:25.290642 (+ 578us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:25.290685 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:25.290703 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:25.290804 (+ 101us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:25.290940 (+ 136us) server_negotiation.cc:300] Negotiation successful
1027 14:56:25.291050 (+ 110us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:56:25.292058 20361 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:25.339792 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.339250 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34378)
1027 14:56:25.339482 (+ 232us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.339595 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20251027 14:56:25.340777 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.340426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34384)
1027 14:56:25.340557 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.340640 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":41}
I20251027 14:56:25.399605 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20239
I20251027 14:56:25.401196 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.400700 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34386)
1027 14:56:25.400913 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.401006 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20251027 14:56:25.402194 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.401867 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34402)
1027 14:56:25.402007 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.402084 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
I20251027 14:56:25.414981 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20369
I20251027 14:56:25.423595 20512 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.415672 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:42076 (local address 127.0.104.3:43259)
1027 14:56:25.415841 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:25.415845 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:25.415864 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:56:25.416028 (+ 164us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:25.416034 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:25.416119 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:25.416236 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:25.416778 (+ 542us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.417715 (+ 937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:25.418796 (+ 1081us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:25.419147 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:25.422311 (+ 3164us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:25.422342 (+ 31us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:25.422358 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:25.422925 (+ 567us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:25.422968 (+ 43us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:25.422986 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:25.423069 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:25.423155 (+ 86us) server_negotiation.cc:300] Negotiation successful
1027 14:56:25.423273 (+ 118us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:56:25.424201 20491 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:25.482677 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.482254 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34406)
1027 14:56:25.482441 (+ 187us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.482528 (+ 87us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
I20251027 14:56:25.483711 20513 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:25.483353 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:37893 (local address 127.0.0.1:34410)
1027 14:56:25.483530 (+ 177us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:25.483593 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:37893: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":79}
I20251027 14:56:25.520030 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20369
I20251027 14:56:25.533571 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20018
I20251027 14:56:25.534735 20079 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:25.643265 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20018
2025-10-27T14:56:25Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/3 (2657 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/4
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:25 dist-test-slave-hxxt krb5kdc[20549](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:25 dist-test-slave-hxxt krb5kdc[20549](info): set up 2 sockets
Oct 27 14:56:25 dist-test-slave-hxxt krb5kdc[20549](info): commencing operation
krb5kdc: starting...
W20251027 14:56:27.532481 416 mini_kdc.cc:121] Time spent starting KDC: real 1.852s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:27 dist-test-slave-hxxt krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:27.804065 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34451
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34451
--encrypt_data_at_rest=true
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:28.127002 20565 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:28.127311 20565 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:28.127440 20565 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:28.135533 20565 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:28.135643 20565 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:28.135726 20565 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:28.135787 20565 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:28.135869 20565 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:28.146349 20565 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34451
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34451
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:28.147935 20565 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:28.149292 20565 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:28.157052 20570 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:28.157080 20571 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:28.157804 20573 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:28.158164 20565 server_base.cc:1047] running on GCE node
I20251027 14:56:28.159016 20565 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:28.160516 20565 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:28.161749 20565 hybrid_clock.cc:648] HybridClock initialized: now 1761576988161672 us; error 52 us; skew 500 ppm
I20251027 14:56:28.162077 20565 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:28 dist-test-slave-hxxt krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:28.165081 20565 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:56:28.166541 20565 webserver.cc:492] Webserver started at http://127.0.104.62:34425/ using document root <none> and password file <none>
I20251027 14:56:28.167301 20565 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:28.167393 20565 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:28.167711 20565 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:28.170255 20565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "1cc9505493664a3c8b62e76bc5b7b168"
format_stamp: "Formatted at 2025-10-27 14:56:28 on dist-test-slave-hxxt"
server_key: "cf580c86b8bd42d8451568b5f45e59dc"
server_key_iv: "5a4a85cfb4f0d627665088b5b110c876"
server_key_version: "encryptionkey@0"
I20251027 14:56:28.170954 20565 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "1cc9505493664a3c8b62e76bc5b7b168"
format_stamp: "Formatted at 2025-10-27 14:56:28 on dist-test-slave-hxxt"
server_key: "cf580c86b8bd42d8451568b5f45e59dc"
server_key_iv: "5a4a85cfb4f0d627665088b5b110c876"
server_key_version: "encryptionkey@0"
I20251027 14:56:28.176107 20565 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251027 14:56:28.179462 20580 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:28.180963 20565 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:28.181133 20565 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "1cc9505493664a3c8b62e76bc5b7b168"
format_stamp: "Formatted at 2025-10-27 14:56:28 on dist-test-slave-hxxt"
server_key: "cf580c86b8bd42d8451568b5f45e59dc"
server_key_iv: "5a4a85cfb4f0d627665088b5b110c876"
server_key_version: "encryptionkey@0"
I20251027 14:56:28.181352 20565 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:28.196774 20565 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:28.201143 20565 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:28.201426 20565 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:28.220080 20565 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34451
I20251027 14:56:28.220124 20631 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34451 every 8 connection(s)
I20251027 14:56:28.221535 20565 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:28.225673 20632 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:28.226871 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20565
I20251027 14:56:28.227067 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:28.227451 416 external_mini_cluster.cc:1468] Setting key e57226ac929768f26f3f429fde7473f6
I20251027 14:56:28.238739 20632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168: Bootstrap starting.
Oct 27 14:56:28 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:28.242429 20632 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:28.243618 20632 log.cc:826] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:28.247012 20632 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168: No bootstrap required, opened a new log
I20251027 14:56:28.249142 20635 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:28.229233 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58248 (local address 127.0.104.62:34451)
1027 14:56:28.229984 (+ 751us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:28.229998 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:28.230052 (+ 54us) server_negotiation.cc:408] Connection header received
1027 14:56:28.231271 (+ 1219us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:28.231301 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:28.231823 (+ 522us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:28.232344 (+ 521us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:28.233800 (+ 1456us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:28.235828 (+ 2028us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:28.237306 (+ 1478us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:28.237880 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:28.241334 (+ 3454us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:28.241427 (+ 93us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:28.241442 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:28.241482 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:28.244709 (+ 3227us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:28.245444 (+ 735us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:28.245453 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:28.245462 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:28.245577 (+ 115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:28.245869 (+ 292us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:28.245875 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:28.245877 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:28.246520 (+ 643us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:28.246836 (+ 316us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:28.247262 (+ 426us) server_negotiation.cc:300] Negotiation successful
1027 14:56:28.247740 (+ 478us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":536,"thread_start_us":302,"threads_started":1}
I20251027 14:56:28.253585 20632 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc9505493664a3c8b62e76bc5b7b168" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34451 } }
I20251027 14:56:28.253897 20632 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:28.253979 20632 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1cc9505493664a3c8b62e76bc5b7b168, State: Initialized, Role: FOLLOWER
I20251027 14:56:28.254488 20632 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc9505493664a3c8b62e76bc5b7b168" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34451 } }
I20251027 14:56:28.254658 20632 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:28.254748 20632 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:28.254856 20632 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:28.257041 20632 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc9505493664a3c8b62e76bc5b7b168" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34451 } }
I20251027 14:56:28.257513 20632 leader_election.cc:304] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1cc9505493664a3c8b62e76bc5b7b168; no voters:
I20251027 14:56:28.258247 20632 leader_election.cc:290] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:28.258333 20637 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:28.259006 20637 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [term 1 LEADER]: Becoming Leader. State: Replica: 1cc9505493664a3c8b62e76bc5b7b168, State: Running, Role: LEADER
I20251027 14:56:28.259495 20637 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc9505493664a3c8b62e76bc5b7b168" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34451 } }
I20251027 14:56:28.260172 20632 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:28.262691 20639 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1cc9505493664a3c8b62e76bc5b7b168. Latest consensus state: current_term: 1 leader_uuid: "1cc9505493664a3c8b62e76bc5b7b168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc9505493664a3c8b62e76bc5b7b168" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34451 } } }
I20251027 14:56:28.263029 20639 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:28.263898 20646 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:28.264046 20638 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1cc9505493664a3c8b62e76bc5b7b168" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1cc9505493664a3c8b62e76bc5b7b168" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34451 } } }
I20251027 14:56:28.265450 20638 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:28.270416 20646 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:28.278955 20646 catalog_manager.cc:1357] Generated new cluster ID: 4fd539b0963448a0afa466ad615fcc1b
I20251027 14:56:28.279075 20646 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:28.312283 20646 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:28.314078 20646 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:28.331005 20646 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168: Generated new TSK 0
I20251027 14:56:28.332085 20646 catalog_manager.cc:1524] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:28.439350 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34451
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:28.775628 20660 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:28.775918 20660 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:28.776050 20660 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:28.784183 20660 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:28.784296 20660 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:28.784504 20660 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:56:28.794972 20660 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:34451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:28.796547 20660 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:28.797834 20660 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:28.806887 20665 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:28.807226 20666 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:28.807564 20668 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:28.808211 20660 server_base.cc:1047] running on GCE node
I20251027 14:56:28.808768 20660 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:28.809623 20660 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:28.810820 20660 hybrid_clock.cc:648] HybridClock initialized: now 1761576988810747 us; error 47 us; skew 500 ppm
I20251027 14:56:28.811085 20660 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:28 dist-test-slave-hxxt krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:28.814154 20660 init.cc:377] Logged in from keytab as kudu/127.0.104.1@KRBTEST.COM (short username kudu)
I20251027 14:56:28.815624 20660 webserver.cc:492] Webserver started at http://127.0.104.1:36935/ using document root <none> and password file <none>
I20251027 14:56:28.816321 20660 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:28.816437 20660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:28.816784 20660 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:28.819329 20660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "48e1162ea6c840aca89962c5d31a16e9"
format_stamp: "Formatted at 2025-10-27 14:56:28 on dist-test-slave-hxxt"
server_key: "73ad0df32b2ae808898dd49ca7e6529a"
server_key_iv: "a87a7dbc13b1f1d09f6aa575ca489278"
server_key_version: "encryptionkey@0"
I20251027 14:56:28.820083 20660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "48e1162ea6c840aca89962c5d31a16e9"
format_stamp: "Formatted at 2025-10-27 14:56:28 on dist-test-slave-hxxt"
server_key: "73ad0df32b2ae808898dd49ca7e6529a"
server_key_iv: "a87a7dbc13b1f1d09f6aa575ca489278"
server_key_version: "encryptionkey@0"
I20251027 14:56:28.825238 20660 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:56:28.828550 20675 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:28.829818 20660 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:56:28.829988 20660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "48e1162ea6c840aca89962c5d31a16e9"
format_stamp: "Formatted at 2025-10-27 14:56:28 on dist-test-slave-hxxt"
server_key: "73ad0df32b2ae808898dd49ca7e6529a"
server_key_iv: "a87a7dbc13b1f1d09f6aa575ca489278"
server_key_version: "encryptionkey@0"
I20251027 14:56:28.830219 20660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:28.889993 20660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:28.893970 20660 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:28.894213 20660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:28.895134 20660 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:28.896656 20660 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:28.896741 20660 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:28.896821 20660 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:28.896873 20660 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:28.926803 20660 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:38535
I20251027 14:56:28.926859 20787 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:38535 every 8 connection(s)
I20251027 14:56:28.928205 20660 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:56:28.933197 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20660
I20251027 14:56:28.933372 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:56:28.933784 416 external_mini_cluster.cc:1468] Setting key 598727d90100c222a3a7feb68dcc78b0
Oct 27 14:56:28 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576988, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.1@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:28.947266 20790 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:28.930789 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:45445 (local address 127.0.104.62:34451)
1027 14:56:28.931204 (+ 415us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:28.931211 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:28.932183 (+ 972us) server_negotiation.cc:408] Connection header received
1027 14:56:28.933738 (+ 1555us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:28.933744 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:28.933833 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:28.933962 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:28.936116 (+ 2154us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:28.937440 (+ 1324us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:28.938536 (+ 1096us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:28.938907 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:28.941690 (+ 2783us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:28.941725 (+ 35us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:28.941730 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:28.941767 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:28.944460 (+ 2693us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:28.945259 (+ 799us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:28.945262 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:28.945265 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:28.945334 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:28.945646 (+ 312us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:28.945649 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:28.945651 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:28.945858 (+ 207us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:28.945978 (+ 120us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:28.946815 (+ 837us) server_negotiation.cc:300] Negotiation successful
1027 14:56:28.947003 (+ 188us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":296,"thread_start_us":187,"threads_started":1}
I20251027 14:56:28.948632 20791 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:28.931327 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34451 (local address 127.0.104.1:45445)
1027 14:56:28.932010 (+ 683us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:28.932053 (+ 43us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:28.933446 (+ 1393us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:28.934151 (+ 705us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:28.934161 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:28.934907 (+ 746us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:28.935839 (+ 932us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:28.935944 (+ 105us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:28.937609 (+ 1665us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:28.937613 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:28.938403 (+ 790us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:28.938415 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:28.938627 (+ 212us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:28.939281 (+ 654us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:28.939306 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:28.941496 (+ 2190us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:28.944617 (+ 3121us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:28.944627 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:28.944646 (+ 19us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:28.945159 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:28.945414 (+ 255us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:28.945419 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:28.945422 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:28.945565 (+ 143us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:28.945955 (+ 390us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:28.945964 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:28.946380 (+ 416us) client_negotiation.cc:769] Sending connection context
1027 14:56:28.946765 (+ 385us) client_negotiation.cc:240] Negotiation successful
1027 14:56:28.947270 (+ 505us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":465,"thread_start_us":290,"threads_started":1}
I20251027 14:56:28.950259 20788 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34451
I20251027 14:56:28.950601 20788 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:28.951368 20788 heartbeater.cc:507] Master 127.0.104.62:34451 requested a full tablet report, sending...
I20251027 14:56:28.953691 20597 ts_manager.cc:194] Registered new tserver with Master: 48e1162ea6c840aca89962c5d31a16e9 (127.0.104.1:38535)
I20251027 14:56:28.955756 20597 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.1@KRBTEST.COM'} at 127.0.104.1:45445
WARNING: no policy specified for kudu/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.2@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.2@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.2 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:29.029402 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34451
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:29.346515 20796 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:29.346825 20796 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:29.346959 20796 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:29.354905 20796 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:29.355023 20796 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:29.355193 20796 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:56:29.365473 20796 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.2
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:34451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:29.366976 20796 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:29.368304 20796 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:29.377048 20802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:29.377017 20801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:29.377621 20804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:29.378365 20796 server_base.cc:1047] running on GCE node
I20251027 14:56:29.378894 20796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:29.379757 20796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:29.380959 20796 hybrid_clock.cc:648] HybridClock initialized: now 1761576989380896 us; error 32 us; skew 500 ppm
I20251027 14:56:29.381237 20796 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:29 dist-test-slave-hxxt krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:29.384229 20796 init.cc:377] Logged in from keytab as kudu/127.0.104.2@KRBTEST.COM (short username kudu)
I20251027 14:56:29.385783 20796 webserver.cc:492] Webserver started at http://127.0.104.2:44959/ using document root <none> and password file <none>
I20251027 14:56:29.386546 20796 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:29.386631 20796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:29.386940 20796 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:29.389509 20796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "53fc053743074c99a3718c1530705ba1"
format_stamp: "Formatted at 2025-10-27 14:56:29 on dist-test-slave-hxxt"
server_key: "17f18b1d6eb0c90af8c1fcf3f22dca15"
server_key_iv: "e26e3c05440a1e98e679e3333e9ffa08"
server_key_version: "encryptionkey@0"
I20251027 14:56:29.390235 20796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "53fc053743074c99a3718c1530705ba1"
format_stamp: "Formatted at 2025-10-27 14:56:29 on dist-test-slave-hxxt"
server_key: "17f18b1d6eb0c90af8c1fcf3f22dca15"
server_key_iv: "e26e3c05440a1e98e679e3333e9ffa08"
server_key_version: "encryptionkey@0"
I20251027 14:56:29.395598 20796 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:56:29.398976 20811 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:29.400286 20796 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:56:29.400473 20796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "53fc053743074c99a3718c1530705ba1"
format_stamp: "Formatted at 2025-10-27 14:56:29 on dist-test-slave-hxxt"
server_key: "17f18b1d6eb0c90af8c1fcf3f22dca15"
server_key_iv: "e26e3c05440a1e98e679e3333e9ffa08"
server_key_version: "encryptionkey@0"
I20251027 14:56:29.400712 20796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:29.437665 20796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:29.441726 20796 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:29.441948 20796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:29.442867 20796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:29.444311 20796 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:29.444414 20796 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:29.444505 20796 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:29.444567 20796 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:29.474723 20796 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:33685
I20251027 14:56:29.474766 20923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:33685 every 8 connection(s)
I20251027 14:56:29.476006 20796 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:56:29.482125 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20796
I20251027 14:56:29.482314 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:56:29.482674 416 external_mini_cluster.cc:1468] Setting key 3ddba137449ae320d2ebd6d9d807e03f
Oct 27 14:56:29 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.2@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:29.494834 20926 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:29.478566 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:59011 (local address 127.0.104.62:34451)
1027 14:56:29.479009 (+ 443us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:29.479016 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:29.479955 (+ 939us) server_negotiation.cc:408] Connection header received
1027 14:56:29.481480 (+ 1525us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:29.481485 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:29.481579 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:29.481673 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:29.483393 (+ 1720us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:29.484525 (+ 1132us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:29.485602 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:29.486057 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:29.488946 (+ 2889us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:29.488987 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:29.488992 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:29.489028 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:29.492055 (+ 3027us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:29.492755 (+ 700us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:29.492761 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:29.492764 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:29.492845 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:29.493124 (+ 279us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:29.493128 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:29.493131 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:29.493409 (+ 278us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:29.493524 (+ 115us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:29.494393 (+ 869us) server_negotiation.cc:300] Negotiation successful
1027 14:56:29.494601 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":327,"thread_start_us":215,"threads_started":1}
I20251027 14:56:29.496083 20927 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:29.479132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34451 (local address 127.0.104.2:59011)
1027 14:56:29.479801 (+ 669us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:29.479837 (+ 36us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:29.481200 (+ 1363us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:29.481860 (+ 660us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:29.481869 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:29.482478 (+ 609us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:29.483197 (+ 719us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:29.483222 (+ 25us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:29.484680 (+ 1458us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:29.484684 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:29.485464 (+ 780us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:29.485474 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:29.485671 (+ 197us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:29.486422 (+ 751us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:29.486454 (+ 32us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:29.488770 (+ 2316us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:29.492190 (+ 3420us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:29.492197 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:29.492211 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:29.492647 (+ 436us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:29.492936 (+ 289us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:29.492940 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:29.492942 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:29.493040 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:29.493521 (+ 481us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:29.493529 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:29.493936 (+ 407us) client_negotiation.cc:769] Sending connection context
1027 14:56:29.494337 (+ 401us) client_negotiation.cc:240] Negotiation successful
1027 14:56:29.494803 (+ 466us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":294,"threads_started":1}
I20251027 14:56:29.497512 20924 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34451
I20251027 14:56:29.497839 20924 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:29.498592 20924 heartbeater.cc:507] Master 127.0.104.62:34451 requested a full tablet report, sending...
I20251027 14:56:29.500041 20597 ts_manager.cc:194] Registered new tserver with Master: 53fc053743074c99a3718c1530705ba1 (127.0.104.2:33685)
I20251027 14:56:29.501076 20597 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.2@KRBTEST.COM'} at 127.0.104.2:59011
WARNING: no policy specified for kudu/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.3@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.3@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.3 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:29.576494 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:34451
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:45591
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:29.899075 20932 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:29.899376 20932 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:29.899515 20932 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:29.907532 20932 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:29.907636 20932 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:29.907810 20932 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:56:29.917955 20932 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:45591
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.3
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.104.62:34451
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:29.919550 20932 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:29.920899 20932 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:29.930060 20937 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:29.930761 20938 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:29.931463 20940 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:29.932479 20932 server_base.cc:1047] running on GCE node
I20251027 14:56:29.933049 20932 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:29.933902 20932 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:29.935115 20932 hybrid_clock.cc:648] HybridClock initialized: now 1761576989935047 us; error 39 us; skew 500 ppm
I20251027 14:56:29.935385 20932 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:29 dist-test-slave-hxxt krb5kdc[20549](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:29.938428 20932 init.cc:377] Logged in from keytab as kudu/127.0.104.3@KRBTEST.COM (short username kudu)
I20251027 14:56:29.939978 20932 webserver.cc:492] Webserver started at http://127.0.104.3:42291/ using document root <none> and password file <none>
I20251027 14:56:29.940717 20932 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:29.940795 20932 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:29.941123 20932 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:29.943532 20932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "8945c0ce61234bd6b3837db8f440b8f1"
format_stamp: "Formatted at 2025-10-27 14:56:29 on dist-test-slave-hxxt"
server_key: "6b25f99ea01b1b9cdd40943619ec7936"
server_key_iv: "9261ab03c49e9df43cd47d9ba96a43cf"
server_key_version: "encryptionkey@0"
I20251027 14:56:29.944242 20932 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "8945c0ce61234bd6b3837db8f440b8f1"
format_stamp: "Formatted at 2025-10-27 14:56:29 on dist-test-slave-hxxt"
server_key: "6b25f99ea01b1b9cdd40943619ec7936"
server_key_iv: "9261ab03c49e9df43cd47d9ba96a43cf"
server_key_version: "encryptionkey@0"
I20251027 14:56:29.949311 20932 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:56:29.952499 20947 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:29.953768 20932 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:56:29.953907 20932 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "8945c0ce61234bd6b3837db8f440b8f1"
format_stamp: "Formatted at 2025-10-27 14:56:29 on dist-test-slave-hxxt"
server_key: "6b25f99ea01b1b9cdd40943619ec7936"
server_key_iv: "9261ab03c49e9df43cd47d9ba96a43cf"
server_key_version: "encryptionkey@0"
I20251027 14:56:29.954079 20932 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:29.958693 20788 heartbeater.cc:499] Master 127.0.104.62:34451 was elected leader, sending a full tablet report...
I20251027 14:56:29.973937 20932 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:29.978222 20932 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:29.978482 20932 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:29.979430 20932 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:29.981005 20932 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:29.981104 20932 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:29.981209 20932 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:29.981276 20932 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:30.012094 20932 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:34965
I20251027 14:56:30.012138 21059 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:34965 every 8 connection(s)
I20251027 14:56:30.013527 20932 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:56:30.017750 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 20932
I20251027 14:56:30.017928 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:56:30.018307 416 external_mini_cluster.cc:1468] Setting key 410fd3b48a3131b6f76abe1c33c6531c
Oct 27 14:56:30 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576989, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.3@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:56:30.031741 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.016172 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:50087 (local address 127.0.104.62:34451)
1027 14:56:30.016596 (+ 424us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.016601 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.017695 (+ 1094us) server_negotiation.cc:408] Connection header received
1027 14:56:30.019379 (+ 1684us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.019384 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.019465 (+ 81us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.019566 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:30.021450 (+ 1884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.022362 (+ 912us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.023430 (+ 1068us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.023892 (+ 462us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.026565 (+ 2673us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:30.026598 (+ 33us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:30.026602 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:30.026634 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:30.029203 (+ 2569us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.029807 (+ 604us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.029811 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.029814 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.029882 (+ 68us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.030123 (+ 241us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.030126 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.030128 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.030358 (+ 230us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:30.030472 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.031206 (+ 734us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.031428 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":325,"thread_start_us":214,"threads_started":1}
I20251027 14:56:30.032776 21063 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.016751 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:34451 (local address 127.0.104.3:50087)
1027 14:56:30.017518 (+ 767us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:30.017565 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:30.019096 (+ 1531us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:30.019838 (+ 742us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:30.019847 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:30.020538 (+ 691us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:30.021283 (+ 745us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.021304 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.022519 (+ 1215us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.022523 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:30.023297 (+ 774us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.023308 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.023569 (+ 261us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.024262 (+ 693us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:56:30.024285 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:30.026377 (+ 2092us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:30.029310 (+ 2933us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:30.029316 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:30.029330 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:30.029710 (+ 380us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:30.029952 (+ 242us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:56:30.029956 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:56:30.029958 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:56:30.030054 (+ 96us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:56:30.030442 (+ 388us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:30.030447 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:30.030763 (+ 316us) client_negotiation.cc:769] Sending connection context
1027 14:56:30.031043 (+ 280us) client_negotiation.cc:240] Negotiation successful
1027 14:56:30.031493 (+ 450us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":510,"thread_start_us":340,"threads_started":1}
I20251027 14:56:30.034050 21060 heartbeater.cc:344] Connected to a master server at 127.0.104.62:34451
I20251027 14:56:30.034365 21060 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:30.035188 21060 heartbeater.cc:507] Master 127.0.104.62:34451 requested a full tablet report, sending...
I20251027 14:56:30.036639 20597 ts_manager.cc:194] Registered new tserver with Master: 8945c0ce61234bd6b3837db8f440b8f1 (127.0.104.3:34965)
I20251027 14:56:30.037434 20597 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.104.3@KRBTEST.COM'} at 127.0.104.3:50087
I20251027 14:56:30.045841 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:56:30.060983 21062 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.049798 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58250 (local address 127.0.104.62:34451)
1027 14:56:30.050008 (+ 210us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.050040 (+ 32us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.050249 (+ 209us) server_negotiation.cc:408] Connection header received
1027 14:56:30.050394 (+ 145us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.050399 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.050478 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.050557 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:30.052147 (+ 1590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.053059 (+ 912us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.054197 (+ 1138us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.054629 (+ 432us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.056205 (+ 1576us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:30.056239 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:30.056242 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:30.056270 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:30.058769 (+ 2499us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.059409 (+ 640us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.059413 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.059415 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.059484 (+ 69us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.059745 (+ 261us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.059748 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.059750 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.060041 (+ 291us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:30.060226 (+ 185us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.060502 (+ 276us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.060704 (+ 202us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":66}
I20251027 14:56:30.064646 20597 catalog_manager.cc:2257] Servicing CreateTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:56:30.069252 20597 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:56:30.103667 21073 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.081639 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:38535 (local address 127.0.0.1:58218)
1027 14:56:30.082410 (+ 771us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:30.082463 (+ 53us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:30.082653 (+ 190us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:30.097823 (+ 15170us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:30.097833 (+ 10us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:30.097875 (+ 42us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:30.098903 (+ 1028us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.098920 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.101169 (+ 2249us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.101174 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:30.102561 (+ 1387us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.102573 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.102786 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.102870 (+ 84us) client_negotiation.cc:769] Sending connection context
1027 14:56:30.103038 (+ 168us) client_negotiation.cc:240] Negotiation successful
1027 14:56:30.103237 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":531,"thread_start_us":375,"threads_started":1}
I20251027 14:56:30.104514 21074 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.081943 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58218 (local address 127.0.104.1:38535)
1027 14:56:30.097094 (+ 15151us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.097104 (+ 10us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.097134 (+ 30us) server_negotiation.cc:408] Connection header received
1027 14:56:30.097239 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.097246 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.097537 (+ 291us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.097819 (+ 282us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:30.099114 (+ 1295us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.100982 (+ 1868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.102715 (+ 1733us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.103745 (+ 1030us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.103906 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.104036 (+ 130us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.104213 (+ 177us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15019,"thread_start_us":14920,"threads_started":1}
I20251027 14:56:30.104655 21076 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.082421 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34965 (local address 127.0.0.1:58704)
1027 14:56:30.096263 (+ 13842us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:30.096285 (+ 22us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:30.096483 (+ 198us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:30.098614 (+ 2131us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:30.098619 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:30.098649 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:30.099249 (+ 600us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.099259 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.101867 (+ 2608us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.101873 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:30.103751 (+ 1878us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.103766 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.104102 (+ 336us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.104130 (+ 28us) client_negotiation.cc:769] Sending connection context
1027 14:56:30.104192 (+ 62us) client_negotiation.cc:240] Negotiation successful
1027 14:56:30.104316 (+ 124us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":13725,"thread_start_us":15851,"threads_started":1}
I20251027 14:56:30.105851 21075 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.083073 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33685 (local address 127.0.0.1:49744)
1027 14:56:30.098630 (+ 15557us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:30.098650 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:30.098879 (+ 229us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:30.099874 (+ 995us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:30.099879 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:30.099906 (+ 27us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:30.100669 (+ 763us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.100683 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.102634 (+ 1951us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.102640 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:30.104299 (+ 1659us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.104311 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.105454 (+ 1143us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.105472 (+ 18us) client_negotiation.cc:769] Sending connection context
1027 14:56:30.105565 (+ 93us) client_negotiation.cc:240] Negotiation successful
1027 14:56:30.105649 (+ 84us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":15422,"thread_start_us":12950,"threads_started":1}
I20251027 14:56:30.106771 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.083005 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58704 (local address 127.0.104.3:34965)
1027 14:56:30.097881 (+ 14876us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.097889 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.097924 (+ 35us) server_negotiation.cc:408] Connection header received
1027 14:56:30.098025 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.098034 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.098292 (+ 258us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.098528 (+ 236us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:30.099412 (+ 884us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.101637 (+ 2225us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.104997 (+ 3360us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.106051 (+ 1054us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.106202 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.106325 (+ 123us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.106507 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":14720,"thread_start_us":14574,"threads_started":1}
I20251027 14:56:30.107067 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.083365 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49744 (local address 127.0.104.2:33685)
1027 14:56:30.095632 (+ 12267us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.095644 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.098771 (+ 3127us) server_negotiation.cc:408] Connection header received
1027 14:56:30.099081 (+ 310us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.099090 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.099326 (+ 236us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.099614 (+ 288us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:30.100857 (+ 1243us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.102358 (+ 1501us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.104499 (+ 2141us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.106484 (+ 1985us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.106574 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.106707 (+ 133us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.106830 (+ 123us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":12081,"thread_start_us":12287,"threads_started":1}
I20251027 14:56:30.109822 20723 tablet_service.cc:1505] Processing CreateTablet for tablet b2acf9f2ee9e44599a4e09190fe87e70 (DEFAULT_TABLE table=test-table [id=0de40c273b8e4e1faec9c31a09293f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:30.111083 20723 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2acf9f2ee9e44599a4e09190fe87e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:30.110934 20995 tablet_service.cc:1505] Processing CreateTablet for tablet b2acf9f2ee9e44599a4e09190fe87e70 (DEFAULT_TABLE table=test-table [id=0de40c273b8e4e1faec9c31a09293f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:30.110934 20859 tablet_service.cc:1505] Processing CreateTablet for tablet b2acf9f2ee9e44599a4e09190fe87e70 (DEFAULT_TABLE table=test-table [id=0de40c273b8e4e1faec9c31a09293f1f]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:30.112327 20995 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2acf9f2ee9e44599a4e09190fe87e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:30.112337 20859 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2acf9f2ee9e44599a4e09190fe87e70. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:30.122354 21079 tablet_bootstrap.cc:492] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Bootstrap starting.
I20251027 14:56:30.125982 21079 tablet_bootstrap.cc:654] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:30.125953 21080 tablet_bootstrap.cc:492] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Bootstrap starting.
I20251027 14:56:30.127879 21079 log.cc:826] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:30.129475 21080 tablet_bootstrap.cc:654] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:30.129455 21081 tablet_bootstrap.cc:492] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Bootstrap starting.
I20251027 14:56:30.130781 21080 log.cc:826] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:30.131234 21079 tablet_bootstrap.cc:492] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: No bootstrap required, opened a new log
I20251027 14:56:30.131570 21079 ts_tablet_manager.cc:1403] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Time spent bootstrapping tablet: real 0.009s user 0.005s sys 0.002s
I20251027 14:56:30.133054 21081 tablet_bootstrap.cc:654] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:30.134025 21080 tablet_bootstrap.cc:492] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: No bootstrap required, opened a new log
I20251027 14:56:30.134305 21080 ts_tablet_manager.cc:1403] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Time spent bootstrapping tablet: real 0.009s user 0.006s sys 0.000s
I20251027 14:56:30.134315 21081 log.cc:826] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:30.137434 21079 raft_consensus.cc:359] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.137784 21079 raft_consensus.cc:385] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:30.137888 21079 raft_consensus.cc:740] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48e1162ea6c840aca89962c5d31a16e9, State: Initialized, Role: FOLLOWER
I20251027 14:56:30.137939 21081 tablet_bootstrap.cc:492] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: No bootstrap required, opened a new log
I20251027 14:56:30.138253 21081 ts_tablet_manager.cc:1403] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Time spent bootstrapping tablet: real 0.009s user 0.008s sys 0.000s
I20251027 14:56:30.138511 21079 consensus_queue.cc:260] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.141388 21079 ts_tablet_manager.cc:1434] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Time spent starting tablet: real 0.010s user 0.006s sys 0.002s
I20251027 14:56:30.141181 21080 raft_consensus.cc:359] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.141536 21080 raft_consensus.cc:385] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:30.141701 21080 raft_consensus.cc:740] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 53fc053743074c99a3718c1530705ba1, State: Initialized, Role: FOLLOWER
I20251027 14:56:30.142371 21080 consensus_queue.cc:260] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.143417 20924 heartbeater.cc:499] Master 127.0.104.62:34451 was elected leader, sending a full tablet report...
I20251027 14:56:30.143997 21080 ts_tablet_manager.cc:1434] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Time spent starting tablet: real 0.010s user 0.010s sys 0.000s
I20251027 14:56:30.145030 21081 raft_consensus.cc:359] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.145435 21081 raft_consensus.cc:385] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:30.145546 21081 raft_consensus.cc:740] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8945c0ce61234bd6b3837db8f440b8f1, State: Initialized, Role: FOLLOWER
I20251027 14:56:30.146143 21081 consensus_queue.cc:260] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.147537 21060 heartbeater.cc:499] Master 127.0.104.62:34451 was elected leader, sending a full tablet report...
I20251027 14:56:30.148418 21081 ts_tablet_manager.cc:1434] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Time spent starting tablet: real 0.010s user 0.009s sys 0.000s
W20251027 14:56:30.180624 20789 tablet.cc:2378] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:30.228035 20925 tablet.cc:2378] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20251027 14:56:30.265111 21061 tablet.cc:2378] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:56:30.529238 21085 raft_consensus.cc:493] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:30.529510 21085 raft_consensus.cc:515] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.531337 21085 leader_election.cc:290] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 53fc053743074c99a3718c1530705ba1 (127.0.104.2:33685), 8945c0ce61234bd6b3837db8f440b8f1 (127.0.104.3:34965)
I20251027 14:56:30.537029 21088 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.531686 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:34965 (local address 127.0.104.1:36407)
1027 14:56:30.532366 (+ 680us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:30.532410 (+ 44us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:30.532564 (+ 154us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:30.532911 (+ 347us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:30.532916 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:30.532940 (+ 24us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:30.533751 (+ 811us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.533764 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.535079 (+ 1315us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.535083 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:30.536275 (+ 1192us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.536286 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.536458 (+ 172us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.536489 (+ 31us) client_negotiation.cc:769] Sending connection context
1027 14:56:30.536573 (+ 84us) client_negotiation.cc:240] Negotiation successful
1027 14:56:30.536692 (+ 119us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":565,"thread_start_us":329,"threads_started":1}
I20251027 14:56:30.537068 21089 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.531727 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:33685 (local address 127.0.104.1:52941)
1027 14:56:30.532486 (+ 759us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:30.532503 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:30.532641 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:30.532941 (+ 300us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:30.532945 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:30.532968 (+ 23us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:30.533766 (+ 798us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.533777 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.535032 (+ 1255us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.535036 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:30.536302 (+ 1266us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:30.536314 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.536530 (+ 216us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.536554 (+ 24us) client_negotiation.cc:769] Sending connection context
1027 14:56:30.536620 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:56:30.536736 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":670,"thread_start_us":499,"threads_started":1}
I20251027 14:56:30.537813 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.531766 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:36407 (local address 127.0.104.3:34965)
1027 14:56:30.531945 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.531952 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.532513 (+ 561us) server_negotiation.cc:408] Connection header received
1027 14:56:30.532722 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.532729 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.532804 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.532869 (+ 65us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:30.533896 (+ 1027us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.534843 (+ 947us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.536440 (+ 1597us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.537436 (+ 996us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.537490 (+ 54us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.537554 (+ 64us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.537643 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:56:30.537863 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.531779 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:52941 (local address 127.0.104.2:33685)
1027 14:56:30.532091 (+ 312us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.532097 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.532605 (+ 508us) server_negotiation.cc:408] Connection header received
1027 14:56:30.532757 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.532762 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.532829 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.532914 (+ 85us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:30.533902 (+ 988us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.534832 (+ 930us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.536465 (+ 1633us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.537493 (+ 1028us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.537544 (+ 51us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.537605 (+ 61us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.537694 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":196}
I20251027 14:56:30.538657 20879 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2acf9f2ee9e44599a4e09190fe87e70" candidate_uuid: "48e1162ea6c840aca89962c5d31a16e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53fc053743074c99a3718c1530705ba1" is_pre_election: true
I20251027 14:56:30.538646 21015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2acf9f2ee9e44599a4e09190fe87e70" candidate_uuid: "48e1162ea6c840aca89962c5d31a16e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8945c0ce61234bd6b3837db8f440b8f1" is_pre_election: true
I20251027 14:56:30.539069 20879 raft_consensus.cc:2468] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e1162ea6c840aca89962c5d31a16e9 in term 0.
I20251027 14:56:30.539072 21015 raft_consensus.cc:2468] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 48e1162ea6c840aca89962c5d31a16e9 in term 0.
I20251027 14:56:30.539619 20677 leader_election.cc:304] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 48e1162ea6c840aca89962c5d31a16e9, 53fc053743074c99a3718c1530705ba1; no voters:
I20251027 14:56:30.539978 21085 raft_consensus.cc:2804] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:56:30.540127 21085 raft_consensus.cc:493] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:56:30.540320 21085 raft_consensus.cc:3060] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:30.542665 21085 raft_consensus.cc:515] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.543263 21085 leader_election.cc:290] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [CANDIDATE]: Term 1 election: Requested vote from peers 53fc053743074c99a3718c1530705ba1 (127.0.104.2:33685), 8945c0ce61234bd6b3837db8f440b8f1 (127.0.104.3:34965)
I20251027 14:56:30.543622 20879 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2acf9f2ee9e44599a4e09190fe87e70" candidate_uuid: "48e1162ea6c840aca89962c5d31a16e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "53fc053743074c99a3718c1530705ba1"
I20251027 14:56:30.543763 21015 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "b2acf9f2ee9e44599a4e09190fe87e70" candidate_uuid: "48e1162ea6c840aca89962c5d31a16e9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8945c0ce61234bd6b3837db8f440b8f1"
I20251027 14:56:30.543820 20879 raft_consensus.cc:3060] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:30.543943 21015 raft_consensus.cc:3060] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:30.547178 20879 raft_consensus.cc:2468] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e1162ea6c840aca89962c5d31a16e9 in term 1.
I20251027 14:56:30.547181 21015 raft_consensus.cc:2468] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 48e1162ea6c840aca89962c5d31a16e9 in term 1.
I20251027 14:56:30.547642 20677 leader_election.cc:304] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 48e1162ea6c840aca89962c5d31a16e9, 53fc053743074c99a3718c1530705ba1, 8945c0ce61234bd6b3837db8f440b8f1; no voters:
I20251027 14:56:30.547967 21085 raft_consensus.cc:2804] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:30.548419 21085 raft_consensus.cc:697] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 1 LEADER]: Becoming Leader. State: Replica: 48e1162ea6c840aca89962c5d31a16e9, State: Running, Role: LEADER
I20251027 14:56:30.548916 21085 consensus_queue.cc:237] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } }
I20251027 14:56:30.553458 20596 catalog_manager.cc:5649] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 reported cstate change: term changed from 0 to 1, leader changed from <none> to 48e1162ea6c840aca89962c5d31a16e9 (127.0.104.1). New cstate: current_term: 1 leader_uuid: "48e1162ea6c840aca89962c5d31a16e9" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "48e1162ea6c840aca89962c5d31a16e9" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 38535 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 } health_report { overall_health: UNKNOWN } } }
I20251027 14:56:30.671489 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.666130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58228 (local address 127.0.104.1:38535)
1027 14:56:30.666811 (+ 681us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.666817 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.666833 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:56:30.666919 (+ 86us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.666923 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.666997 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.667128 (+ 131us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:56:30.667817 (+ 689us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.668731 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.669949 (+ 1218us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.670252 (+ 303us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.670398 (+ 146us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:56:30.670932 (+ 534us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:56:30.671040 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.671135 (+ 95us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.671215 (+ 80us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":594,"thread_start_us":509,"threads_started":1}
I20251027 14:56:30.679316 20879 raft_consensus.cc:1275] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 1 FOLLOWER]: Refusing update from remote peer 48e1162ea6c840aca89962c5d31a16e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:30.679327 21015 raft_consensus.cc:1275] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 1 FOLLOWER]: Refusing update from remote peer 48e1162ea6c840aca89962c5d31a16e9: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:30.680368 21085 consensus_queue.cc:1048] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "53fc053743074c99a3718c1530705ba1" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 33685 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:30.680661 21090 consensus_queue.cc:1048] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8945c0ce61234bd6b3837db8f440b8f1" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 34965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:30.690478 21095 mvcc.cc:204] Tried to move back new op lower bound from 7215419353811156992 to 7215419353292353536. Current Snapshot: MvccSnapshot[applied={T|T < 7215419353811156992}]
I20251027 14:56:30.707073 20597 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:30.707312 20597 catalog_manager.cc:2755] Servicing DeleteTable request from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:58250:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:30.710716 20597 catalog_manager.cc:5936] T 00000000000000000000000000000000 P 1cc9505493664a3c8b62e76bc5b7b168: Sending DeleteTablet for 3 replicas of tablet b2acf9f2ee9e44599a4e09190fe87e70
I20251027 14:56:30.711884 20723 tablet_service.cc:1552] Processing DeleteTablet for tablet b2acf9f2ee9e44599a4e09190fe87e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:30 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:58218
I20251027 14:56:30.712087 20859 tablet_service.cc:1552] Processing DeleteTablet for tablet b2acf9f2ee9e44599a4e09190fe87e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:30 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:49744
I20251027 14:56:30.712095 20995 tablet_service.cc:1552] Processing DeleteTablet for tablet b2acf9f2ee9e44599a4e09190fe87e70 with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:30 UTC) from {username='kudu', principal='kudu/127.0.104.62@KRBTEST.COM'} at 127.0.0.1:58704
I20251027 14:56:30.713922 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20660
I20251027 14:56:30.724462 21107 tablet_replica.cc:333] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: stopping tablet replica
I20251027 14:56:30.725114 21107 raft_consensus.cc:2243] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:56:30.725661 21108 tablet_replica.cc:333] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: stopping tablet replica
I20251027 14:56:30.725937 21107 raft_consensus.cc:2272] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:30.726205 21108 raft_consensus.cc:2243] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:30.726733 21108 raft_consensus.cc:2272] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:30.727627 21109 tablet_replica.cc:333] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: stopping tablet replica
I20251027 14:56:30.728446 21107 ts_tablet_manager.cc:1916] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:30.729007 21108 ts_tablet_manager.cc:1916] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:30.735900 21109 raft_consensus.cc:2243] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:30.736526 21109 raft_consensus.cc:2272] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:30.736613 21107 ts_tablet_manager.cc:1929] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:30.736821 21107 log.cc:1199] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-0/wal/wals/b2acf9f2ee9e44599a4e09190fe87e70
I20251027 14:56:30.737452 21107 ts_tablet_manager.cc:1950] T b2acf9f2ee9e44599a4e09190fe87e70 P 48e1162ea6c840aca89962c5d31a16e9: Deleting consensus metadata
I20251027 14:56:30.738674 21109 ts_tablet_manager.cc:1916] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:30.739524 20582 catalog_manager.cc:4985] TS 48e1162ea6c840aca89962c5d31a16e9 (127.0.104.1:38535): tablet b2acf9f2ee9e44599a4e09190fe87e70 (table test-table [id=0de40c273b8e4e1faec9c31a09293f1f]) successfully deleted
I20251027 14:56:30.740302 21108 ts_tablet_manager.cc:1929] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:30.740487 21108 log.cc:1199] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/b2acf9f2ee9e44599a4e09190fe87e70
I20251027 14:56:30.740963 21108 ts_tablet_manager.cc:1950] T b2acf9f2ee9e44599a4e09190fe87e70 P 53fc053743074c99a3718c1530705ba1: Deleting consensus metadata
I20251027 14:56:30.742681 20582 catalog_manager.cc:4985] TS 53fc053743074c99a3718c1530705ba1 (127.0.104.2:33685): tablet b2acf9f2ee9e44599a4e09190fe87e70 (table test-table [id=0de40c273b8e4e1faec9c31a09293f1f]) successfully deleted
I20251027 14:56:30.744820 21109 ts_tablet_manager.cc:1929] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:30.744949 21109 log.cc:1199] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_4.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/b2acf9f2ee9e44599a4e09190fe87e70
I20251027 14:56:30.745395 21109 ts_tablet_manager.cc:1950] T b2acf9f2ee9e44599a4e09190fe87e70 P 8945c0ce61234bd6b3837db8f440b8f1: Deleting consensus metadata
Oct 27 14:56:30 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.1@KRBTEST.COM
I20251027 14:56:30.747099 20583 catalog_manager.cc:4985] TS 8945c0ce61234bd6b3837db8f440b8f1 (127.0.104.3:34965): tablet b2acf9f2ee9e44599a4e09190fe87e70 (table test-table [id=0de40c273b8e4e1faec9c31a09293f1f]) successfully deleted
I20251027 14:56:30.753870 21094 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.715146 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58230 (local address 127.0.104.1:38535)
1027 14:56:30.721622 (+ 6476us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.721628 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.730525 (+ 8897us) server_negotiation.cc:408] Connection header received
1027 14:56:30.731022 (+ 497us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.731029 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.731148 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.731225 (+ 77us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:30.733452 (+ 2227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.734965 (+ 1513us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.736836 (+ 1871us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.737348 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.746980 (+ 9632us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:30.747036 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:30.747055 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:30.747109 (+ 54us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:30.751190 (+ 4081us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.751881 (+ 691us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.751891 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.751894 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.751998 (+ 104us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.752288 (+ 290us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.752293 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.752297 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.752692 (+ 395us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:30.752988 (+ 296us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.753238 (+ 250us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.753472 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":6339}
I20251027 14:56:30.754513 20783 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:30.851274 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20660
I20251027 14:56:30.866583 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20796
Oct 27 14:56:30 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.2@KRBTEST.COM
I20251027 14:56:30.881345 21078 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.867387 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34914 (local address 127.0.104.2:33685)
1027 14:56:30.867610 (+ 223us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.867616 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.867638 (+ 22us) server_negotiation.cc:408] Connection header received
1027 14:56:30.867702 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.867707 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.867778 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.867870 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:30.869338 (+ 1468us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.870252 (+ 914us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.871412 (+ 1160us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.871834 (+ 422us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.875392 (+ 3558us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:30.875429 (+ 37us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:30.875445 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:30.875486 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:30.878272 (+ 2786us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.879049 (+ 777us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.879059 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.879062 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.879176 (+ 114us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.879504 (+ 328us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.879510 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.879514 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.880048 (+ 534us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:30.880427 (+ 379us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.880801 (+ 374us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.881026 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:56:30.882287 20919 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:30.970533 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20796
I20251027 14:56:30.984593 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20932
Oct 27 14:56:30 dist-test-slave-hxxt krb5kdc[20549](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576987, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.3@KRBTEST.COM
I20251027 14:56:30.998953 21077 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:30.985357 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33224 (local address 127.0.104.3:34965)
1027 14:56:30.985520 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:30.985527 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:30.985546 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:56:30.985737 (+ 191us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:30.985742 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:30.985816 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:30.985924 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:30.987385 (+ 1461us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.988284 (+ 899us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:30.989519 (+ 1235us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:30.989944 (+ 425us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:30.993635 (+ 3691us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:30.993678 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:30.993694 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:30.993731 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:30.996487 (+ 2756us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.997122 (+ 635us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.997130 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.997132 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.997220 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:30.997541 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:30.997547 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:30.997550 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:30.997828 (+ 278us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:30.998154 (+ 326us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:30.998383 (+ 229us) server_negotiation.cc:300] Negotiation successful
1027 14:56:30.998631 (+ 248us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":52}
I20251027 14:56:30.999609 21055 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:31.088245 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20932
I20251027 14:56:31.102018 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 20565
I20251027 14:56:31.103099 20627 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:31.212275 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 20565
2025-10-27T14:56:31Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/4 (5569 ms)
[ RUN ] EncryptionPolicyTest.TestEncryptionPolicy/5
2025-10-27T14:56:31Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:31Z Disabled control of system clock
I20251027 14:56:31.283555 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43083
--webserver_interface=127.0.104.62
--webserver_port=0
--builtin_ntp_servers=127.0.104.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:43083
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251027 14:56:31.603302 21118 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:31.603621 21118 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:31.603752 21118 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:31.611905 21118 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:31.612015 21118 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:31.612071 21118 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:31.612133 21118 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:31.612219 21118 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:31.622124 21118 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:43083
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:43083
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:31.623715 21118 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:31.625464 21118 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:31.632941 21124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:31.633021 21123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:31.633299 21126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:31.634091 21118 server_base.cc:1047] running on GCE node
I20251027 14:56:31.635010 21118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:31.636404 21118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:31.637650 21118 hybrid_clock.cc:648] HybridClock initialized: now 1761576991637585 us; error 116 us; skew 500 ppm
I20251027 14:56:31.638010 21118 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:31.640126 21118 webserver.cc:492] Webserver started at http://127.0.104.62:44143/ using document root <none> and password file <none>
I20251027 14:56:31.640874 21118 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:31.640988 21118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:31.641319 21118 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:31.643929 21118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "f2a972ff5003467da37e3d31e00a5fcc"
format_stamp: "Formatted at 2025-10-27 14:56:31 on dist-test-slave-hxxt"
server_key: "52c9c3484f453a390535af7153c8c6c4"
server_key_iv: "c13459917f3eeffebad40ba52226c565"
server_key_version: "encryptionkey@0"
I20251027 14:56:31.644703 21118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "f2a972ff5003467da37e3d31e00a5fcc"
format_stamp: "Formatted at 2025-10-27 14:56:31 on dist-test-slave-hxxt"
server_key: "52c9c3484f453a390535af7153c8c6c4"
server_key_iv: "c13459917f3eeffebad40ba52226c565"
server_key_version: "encryptionkey@0"
I20251027 14:56:31.649942 21118 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20251027 14:56:31.653513 21132 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:31.654834 21118 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20251027 14:56:31.655007 21118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "f2a972ff5003467da37e3d31e00a5fcc"
format_stamp: "Formatted at 2025-10-27 14:56:31 on dist-test-slave-hxxt"
server_key: "52c9c3484f453a390535af7153c8c6c4"
server_key_iv: "c13459917f3eeffebad40ba52226c565"
server_key_version: "encryptionkey@0"
I20251027 14:56:31.655210 21118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:31.685379 21118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:31.686220 21118 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:31.686479 21118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:31.704576 21118 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:43083
I20251027 14:56:31.704598 21183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:43083 every 8 connection(s)
I20251027 14:56:31.706010 21118 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:31.710125 21184 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:31.715788 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21118
I20251027 14:56:31.715965 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:31.716336 416 external_mini_cluster.cc:1468] Setting key 78e3e962656f10132f1f855b79e2ecee
I20251027 14:56:31.720795 21184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc: Bootstrap starting.
I20251027 14:56:31.724352 21184 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:31.725750 21184 log.cc:826] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:31.728940 21187 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:31.718206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47182 (local address 127.0.104.62:43083)
1027 14:56:31.719298 (+ 1092us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:31.719314 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:31.719362 (+ 48us) server_negotiation.cc:408] Connection header received
1027 14:56:31.720560 (+ 1198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:31.720584 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:31.721100 (+ 516us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:31.721462 (+ 362us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:31.722405 (+ 943us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:31.723769 (+ 1364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:31.725373 (+ 1604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:31.725857 (+ 484us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:31.726430 (+ 573us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:31.726508 (+ 78us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:31.726529 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:31.727267 (+ 738us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:31.727315 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:31.727334 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:31.727423 (+ 89us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:31.727657 (+ 234us) server_negotiation.cc:300] Negotiation successful
1027 14:56:31.727998 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":623,"thread_start_us":429,"threads_started":1}
I20251027 14:56:31.728976 21184 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc: No bootstrap required, opened a new log
I20251027 14:56:31.734143 21184 raft_consensus.cc:359] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a972ff5003467da37e3d31e00a5fcc" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43083 } }
I20251027 14:56:31.734467 21184 raft_consensus.cc:385] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:31.734561 21184 raft_consensus.cc:740] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f2a972ff5003467da37e3d31e00a5fcc, State: Initialized, Role: FOLLOWER
I20251027 14:56:31.735150 21184 consensus_queue.cc:260] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a972ff5003467da37e3d31e00a5fcc" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43083 } }
I20251027 14:56:31.735365 21184 raft_consensus.cc:399] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:31.735483 21184 raft_consensus.cc:493] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:31.735638 21184 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:31.737908 21184 raft_consensus.cc:515] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a972ff5003467da37e3d31e00a5fcc" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43083 } }
I20251027 14:56:31.738391 21184 leader_election.cc:304] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f2a972ff5003467da37e3d31e00a5fcc; no voters:
I20251027 14:56:31.738940 21184 leader_election.cc:290] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:31.739061 21189 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:31.739648 21189 raft_consensus.cc:697] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [term 1 LEADER]: Becoming Leader. State: Replica: f2a972ff5003467da37e3d31e00a5fcc, State: Running, Role: LEADER
I20251027 14:56:31.740118 21189 consensus_queue.cc:237] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a972ff5003467da37e3d31e00a5fcc" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43083 } }
I20251027 14:56:31.740849 21184 sys_catalog.cc:565] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:31.742830 21191 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [sys.catalog]: SysCatalogTable state changed. Reason: New leader f2a972ff5003467da37e3d31e00a5fcc. Latest consensus state: current_term: 1 leader_uuid: "f2a972ff5003467da37e3d31e00a5fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a972ff5003467da37e3d31e00a5fcc" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43083 } } }
I20251027 14:56:31.743079 21191 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:31.743481 21190 sys_catalog.cc:455] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f2a972ff5003467da37e3d31e00a5fcc" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f2a972ff5003467da37e3d31e00a5fcc" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 43083 } } }
I20251027 14:56:31.743680 21190 sys_catalog.cc:458] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:31.749512 21194 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:31.754235 21194 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:31.762782 21194 catalog_manager.cc:1357] Generated new cluster ID: 282d9d057d044a9b83abe3faf36b537b
I20251027 14:56:31.762887 21194 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:31.802170 21194 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:31.803903 21194 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:31.821056 21194 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc: Generated new TSK 0
I20251027 14:56:31.821902 21194 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:56:31.835206 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.1:0
--local_ip_for_outbound_sockets=127.0.104.1
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43083
--builtin_ntp_servers=127.0.104.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
W20251027 14:56:32.162000 21208 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:32.162281 21208 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:32.162398 21208 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:32.170284 21208 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:32.170368 21208 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:32.170522 21208 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.1
I20251027 14:56:32.179894 21208 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.1:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.104.1
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.1
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:32.181454 21208 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:32.183127 21208 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:32.191871 21214 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:32.191965 21216 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:32.192297 21213 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:32.193045 21208 server_base.cc:1047] running on GCE node
I20251027 14:56:32.193595 21208 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:32.194434 21208 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:32.195637 21208 hybrid_clock.cc:648] HybridClock initialized: now 1761576992195578 us; error 39 us; skew 500 ppm
I20251027 14:56:32.195896 21208 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:32.197964 21208 webserver.cc:492] Webserver started at http://127.0.104.1:44855/ using document root <none> and password file <none>
I20251027 14:56:32.198669 21208 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:32.198784 21208 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:32.199112 21208 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:32.201674 21208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data/instance:
uuid: "a2244e513e4444c0a11b63b92eb7b068"
format_stamp: "Formatted at 2025-10-27 14:56:32 on dist-test-slave-hxxt"
server_key: "42eac675be248a96015287173cc52995"
server_key_iv: "45930a08aa0ddc829a07855868ab917f"
server_key_version: "encryptionkey@0"
I20251027 14:56:32.202410 21208 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal/instance:
uuid: "a2244e513e4444c0a11b63b92eb7b068"
format_stamp: "Formatted at 2025-10-27 14:56:32 on dist-test-slave-hxxt"
server_key: "42eac675be248a96015287173cc52995"
server_key_iv: "45930a08aa0ddc829a07855868ab917f"
server_key_version: "encryptionkey@0"
I20251027 14:56:32.207489 21208 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:56:32.211021 21222 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:32.212453 21208 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:32.212623 21208 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal
uuid: "a2244e513e4444c0a11b63b92eb7b068"
format_stamp: "Formatted at 2025-10-27 14:56:32 on dist-test-slave-hxxt"
server_key: "42eac675be248a96015287173cc52995"
server_key_iv: "45930a08aa0ddc829a07855868ab917f"
server_key_version: "encryptionkey@0"
I20251027 14:56:32.212818 21208 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:32.242383 21208 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:32.243228 21208 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:32.243484 21208 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:32.244406 21208 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:32.245944 21208 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:32.246026 21208 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:32.246140 21208 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:32.246189 21208 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:32.275887 21208 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.1:34825
I20251027 14:56:32.275949 21334 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.1:34825 every 8 connection(s)
I20251027 14:56:32.277288 21208 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/data/info.pb
I20251027 14:56:32.282332 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21208
I20251027 14:56:32.282480 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-0/wal/instance
I20251027 14:56:32.282840 416 external_mini_cluster.cc:1468] Setting key 68c0ec5f940ea0bc2b78ad3d16ef03bf
I20251027 14:56:32.285728 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.2:0
--local_ip_for_outbound_sockets=127.0.104.2
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43083
--builtin_ntp_servers=127.0.104.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251027 14:56:32.289160 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:32.279785 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.1:42295 (local address 127.0.104.62:43083)
1027 14:56:32.280206 (+ 421us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:32.280211 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:32.281461 (+ 1250us) server_negotiation.cc:408] Connection header received
1027 14:56:32.283027 (+ 1566us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:32.283034 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:32.283119 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:32.283250 (+ 131us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:32.284532 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.285810 (+ 1278us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:32.287262 (+ 1452us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.287585 (+ 323us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:32.288415 (+ 830us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:32.288451 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:32.288455 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:32.288471 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:32.288508 (+ 37us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:32.288520 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:32.288588 (+ 68us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:32.288830 (+ 242us) server_negotiation.cc:300] Negotiation successful
1027 14:56:32.288921 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":313,"thread_start_us":215,"threads_started":1}
I20251027 14:56:32.290316 21338 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:32.280417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43083 (local address 127.0.104.1:42295)
1027 14:56:32.281304 (+ 887us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:32.281343 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:32.282723 (+ 1380us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:32.283460 (+ 737us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:32.283472 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:32.283595 (+ 123us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:32.284334 (+ 739us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:32.284351 (+ 17us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.286015 (+ 1664us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:32.286021 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:32.287119 (+ 1098us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:32.287134 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.287396 (+ 262us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:32.288107 (+ 711us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:32.288111 (+ 4us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:32.288148 (+ 37us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:32.288226 (+ 78us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:32.288607 (+ 381us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:32.288616 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:32.288623 (+ 7us) client_negotiation.cc:769] Sending connection context
1027 14:56:32.288830 (+ 207us) client_negotiation.cc:240] Negotiation successful
1027 14:56:32.289171 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":459,"thread_start_us":290,"threads_started":1}
I20251027 14:56:32.291831 21335 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43083
I20251027 14:56:32.292207 21335 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:32.293169 21335 heartbeater.cc:507] Master 127.0.104.62:43083 requested a full tablet report, sending...
I20251027 14:56:32.295403 21149 ts_manager.cc:194] Registered new tserver with Master: a2244e513e4444c0a11b63b92eb7b068 (127.0.104.1:34825)
I20251027 14:56:32.296943 21149 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.1:42295
W20251027 14:56:32.603776 21339 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:32.604092 21339 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:32.604216 21339 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:32.612236 21339 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:32.612339 21339 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:32.612524 21339 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.2
I20251027 14:56:32.622767 21339 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.2:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.104.2
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.2
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:32.624439 21339 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:32.626188 21339 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:32.635155 21344 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:32.635128 21345 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:32.635751 21347 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:32.636319 21339 server_base.cc:1047] running on GCE node
I20251027 14:56:32.636869 21339 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:32.637751 21339 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:32.639003 21339 hybrid_clock.cc:648] HybridClock initialized: now 1761576992638911 us; error 35 us; skew 500 ppm
I20251027 14:56:32.639312 21339 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:32.641448 21339 webserver.cc:492] Webserver started at http://127.0.104.2:40801/ using document root <none> and password file <none>
I20251027 14:56:32.642163 21339 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:32.642262 21339 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:32.642578 21339 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:32.645080 21339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data/instance:
uuid: "7b2ec6dece804303b489e4c0f9d73229"
format_stamp: "Formatted at 2025-10-27 14:56:32 on dist-test-slave-hxxt"
server_key: "37c96fb556daacee65d4278bf52c5990"
server_key_iv: "2fb7e24abdfa6328260f745f923113f4"
server_key_version: "encryptionkey@0"
I20251027 14:56:32.645761 21339 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal/instance:
uuid: "7b2ec6dece804303b489e4c0f9d73229"
format_stamp: "Formatted at 2025-10-27 14:56:32 on dist-test-slave-hxxt"
server_key: "37c96fb556daacee65d4278bf52c5990"
server_key_iv: "2fb7e24abdfa6328260f745f923113f4"
server_key_version: "encryptionkey@0"
I20251027 14:56:32.650943 21339 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:56:32.654443 21353 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:32.655783 21339 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:56:32.655941 21339 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal
uuid: "7b2ec6dece804303b489e4c0f9d73229"
format_stamp: "Formatted at 2025-10-27 14:56:32 on dist-test-slave-hxxt"
server_key: "37c96fb556daacee65d4278bf52c5990"
server_key_iv: "2fb7e24abdfa6328260f745f923113f4"
server_key_version: "encryptionkey@0"
I20251027 14:56:32.656132 21339 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:32.673365 21339 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:32.674214 21339 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:32.674480 21339 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:32.675381 21339 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:32.677008 21339 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:32.677090 21339 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:32.677196 21339 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:32.677242 21339 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:32.708204 21339 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.2:39349
I20251027 14:56:32.708221 21465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.2:39349 every 8 connection(s)
I20251027 14:56:32.709610 21339 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/data/info.pb
I20251027 14:56:32.710709 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21339
I20251027 14:56:32.710889 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal/instance
I20251027 14:56:32.711336 416 external_mini_cluster.cc:1468] Setting key 1de3459f7cf086c44ffe0da1df0673ba
I20251027 14:56:32.714641 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--encrypt_data_at_rest=true
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.104.3:0
--local_ip_for_outbound_sockets=127.0.104.3
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43083
--builtin_ntp_servers=127.0.104.20:46609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_trace_negotiation
--rpc_authentication=disabled with env {}
I20251027 14:56:32.723682 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:32.712699 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.2:35657 (local address 127.0.104.62:43083)
1027 14:56:32.712879 (+ 180us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:32.712885 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:32.714666 (+ 1781us) server_negotiation.cc:408] Connection header received
1027 14:56:32.716818 (+ 2152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:32.716824 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:32.716900 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:32.716993 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:32.718546 (+ 1553us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.719852 (+ 1306us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:32.721396 (+ 1544us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.721879 (+ 483us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:32.722658 (+ 779us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:32.722696 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:32.722700 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:32.722715 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:32.722747 (+ 32us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:32.722759 (+ 12us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:32.722833 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:32.723260 (+ 427us) server_negotiation.cc:300] Negotiation successful
1027 14:56:32.723389 (+ 129us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
I20251027 14:56:32.724776 21468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:32.713419 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43083 (local address 127.0.104.2:35657)
1027 14:56:32.714519 (+ 1100us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:32.714567 (+ 48us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:32.716476 (+ 1909us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:32.717253 (+ 777us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:32.717265 (+ 12us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:32.717391 (+ 126us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:32.718368 (+ 977us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:32.718395 (+ 27us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.720002 (+ 1607us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:32.720008 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:32.721255 (+ 1247us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:32.721270 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:32.721534 (+ 264us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:32.722369 (+ 835us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:32.722372 (+ 3us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:32.722401 (+ 29us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:32.722475 (+ 74us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:32.722850 (+ 375us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:32.722860 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:32.722948 (+ 88us) client_negotiation.cc:769] Sending connection context
1027 14:56:32.723218 (+ 270us) client_negotiation.cc:240] Negotiation successful
1027 14:56:32.723619 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":589,"thread_start_us":387,"threads_started":1}
I20251027 14:56:32.726244 21466 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43083
I20251027 14:56:32.726565 21466 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:32.727370 21466 heartbeater.cc:507] Master 127.0.104.62:43083 requested a full tablet report, sending...
I20251027 14:56:32.728951 21149 ts_manager.cc:194] Registered new tserver with Master: 7b2ec6dece804303b489e4c0f9d73229 (127.0.104.2:39349)
I20251027 14:56:32.729938 21149 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.2:35657
W20251027 14:56:33.034744 21469 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:33.035027 21469 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:33.035120 21469 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:33.043076 21469 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:33.043159 21469 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:33.043282 21469 flags.cc:425] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.104.3
I20251027 14:56:33.052733 21469 tablet_server_runner.cc:78] Tablet server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:46609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal
--rpc_trace_negotiation=true
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.104.3:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.104.3
--webserver_port=0
--tserver_master_addrs=127.0.104.62:43083
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.104.3
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:33.054268 21469 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:33.055941 21469 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:33.064827 21475 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:33.065411 21474 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:33.065490 21477 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:33.066390 21469 server_base.cc:1047] running on GCE node
I20251027 14:56:33.066936 21469 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:33.067811 21469 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:33.069053 21469 hybrid_clock.cc:648] HybridClock initialized: now 1761576993068975 us; error 35 us; skew 500 ppm
I20251027 14:56:33.069326 21469 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20251027 14:56:33.071422 21469 webserver.cc:492] Webserver started at http://127.0.104.3:44369/ using document root <none> and password file <none>
I20251027 14:56:33.072114 21469 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:33.072211 21469 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:33.072571 21469 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:33.075111 21469 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data/instance:
uuid: "9d7ab09d57db4de68dbd9333984c6848"
format_stamp: "Formatted at 2025-10-27 14:56:33 on dist-test-slave-hxxt"
server_key: "0f8d5f969a69f5c5c6837c89a0f8d0c7"
server_key_iv: "a78b32fd885bd148c3ec88d789b8a401"
server_key_version: "encryptionkey@0"
I20251027 14:56:33.075839 21469 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal/instance:
uuid: "9d7ab09d57db4de68dbd9333984c6848"
format_stamp: "Formatted at 2025-10-27 14:56:33 on dist-test-slave-hxxt"
server_key: "0f8d5f969a69f5c5c6837c89a0f8d0c7"
server_key_iv: "a78b32fd885bd148c3ec88d789b8a401"
server_key_version: "encryptionkey@0"
I20251027 14:56:33.081007 21469 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20251027 14:56:33.084213 21483 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:33.085495 21469 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:33.085645 21469 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal
uuid: "9d7ab09d57db4de68dbd9333984c6848"
format_stamp: "Formatted at 2025-10-27 14:56:33 on dist-test-slave-hxxt"
server_key: "0f8d5f969a69f5c5c6837c89a0f8d0c7"
server_key_iv: "a78b32fd885bd148c3ec88d789b8a401"
server_key_version: "encryptionkey@0"
I20251027 14:56:33.085814 21469 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:33.113471 21469 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:33.114346 21469 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:33.114627 21469 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:33.115536 21469 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20251027 14:56:33.117106 21469 ts_tablet_manager.cc:585] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20251027 14:56:33.117190 21469 ts_tablet_manager.cc:531] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:33.117321 21469 ts_tablet_manager.cc:616] Registered 0 tablets
I20251027 14:56:33.117372 21469 ts_tablet_manager.cc:595] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:33.147751 21469 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.3:45173
I20251027 14:56:33.147795 21595 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.3:45173 every 8 connection(s)
I20251027 14:56:33.149163 21469 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/data/info.pb
I20251027 14:56:33.149593 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21469
I20251027 14:56:33.149766 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal/instance
I20251027 14:56:33.150264 416 external_mini_cluster.cc:1468] Setting key 25a775bcb043dfefeca956a38ad2faed
I20251027 14:56:33.160868 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.152085 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:52715 (local address 127.0.104.62:43083)
1027 14:56:33.152284 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.152290 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.153961 (+ 1671us) server_negotiation.cc:408] Connection header received
1027 14:56:33.155461 (+ 1500us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.155466 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.155532 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.155615 (+ 83us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:33.156953 (+ 1338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.157854 (+ 901us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.158943 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.159409 (+ 466us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.159991 (+ 582us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:33.160067 (+ 76us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:33.160071 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:33.160088 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:33.160136 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:33.160153 (+ 17us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:33.160235 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.160543 (+ 308us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.160638 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:56:33.161779 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.152808 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:43083 (local address 127.0.104.3:52715)
1027 14:56:33.153828 (+ 1020us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.153863 (+ 35us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:33.155211 (+ 1348us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:33.155809 (+ 598us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:33.155818 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:33.155911 (+ 93us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:56:33.156803 (+ 892us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.156824 (+ 21us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.158007 (+ 1183us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.158012 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:33.158828 (+ 816us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.158839 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.159039 (+ 200us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.159728 (+ 689us) client_negotiation.cc:623] Initiating SASL PLAIN handshake
1027 14:56:33.159730 (+ 2us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:56:33.159760 (+ 30us) client_negotiation.cc:814] callback for SASL_CB_AUTHNAME
1027 14:56:33.159827 (+ 67us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:56:33.160240 (+ 413us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:56:33.160250 (+ 10us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:56:33.160258 (+ 8us) client_negotiation.cc:769] Sending connection context
1027 14:56:33.160511 (+ 253us) client_negotiation.cc:240] Negotiation successful
1027 14:56:33.160790 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":574,"thread_start_us":383,"threads_started":1}
I20251027 14:56:33.163141 21596 heartbeater.cc:344] Connected to a master server at 127.0.104.62:43083
I20251027 14:56:33.163497 21596 heartbeater.cc:461] Registering TS with master...
I20251027 14:56:33.164417 21596 heartbeater.cc:507] Master 127.0.104.62:43083 requested a full tablet report, sending...
I20251027 14:56:33.165896 21149 ts_manager.cc:194] Registered new tserver with Master: 9d7ab09d57db4de68dbd9333984c6848 (127.0.104.3:45173)
I20251027 14:56:33.166692 21149 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.104.3:52715
I20251027 14:56:33.177898 416 external_mini_cluster.cc:949] 3 TS(s) registered with all masters
I20251027 14:56:33.186833 21337 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.181823 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47186 (local address 127.0.104.62:43083)
1027 14:56:33.181976 (+ 153us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.181980 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.182354 (+ 374us) server_negotiation.cc:408] Connection header received
1027 14:56:33.182493 (+ 139us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.182497 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.182563 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.182650 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:33.183336 (+ 686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.184179 (+ 843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.185508 (+ 1329us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.185851 (+ 343us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.186272 (+ 421us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:33.186296 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:33.186300 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:33.186314 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:33.186354 (+ 40us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:33.186365 (+ 11us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:33.186440 (+ 75us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.186527 (+ 87us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.186613 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:56:33.190436 21149 catalog_manager.cc:2257] Servicing CreateTable request from {username='slave'} at 127.0.0.1:47186:
name: "test-table"
schema {
columns {
name: "key"
type: INT32
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "val"
type: INT32
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
range_schema {
columns {
name: "key"
}
}
}
W20251027 14:56:33.195112 21149 catalog_manager.cc:7011] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table test-table in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20251027 14:56:33.227829 21609 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.206890 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:39349 (local address 127.0.0.1:48254)
1027 14:56:33.219283 (+ 12393us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.219339 (+ 56us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:33.219596 (+ 257us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:33.220865 (+ 1269us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:33.220876 (+ 11us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:33.220915 (+ 39us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:33.221760 (+ 845us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.221776 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.224096 (+ 2320us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.224101 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:33.225683 (+ 1582us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.225696 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.225860 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.225944 (+ 84us) client_negotiation.cc:769] Sending connection context
1027 14:56:33.227315 (+ 1371us) client_negotiation.cc:240] Negotiation successful
1027 14:56:33.227537 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":12087,"thread_start_us":14958,"threads_started":1}
I20251027 14:56:33.228734 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.207231 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48254 (local address 127.0.104.2:39349)
1027 14:56:33.218422 (+ 11191us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.218434 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.219821 (+ 1387us) server_negotiation.cc:408] Connection header received
1027 14:56:33.219944 (+ 123us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.219953 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.220255 (+ 302us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.220516 (+ 261us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:33.221992 (+ 1476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.223759 (+ 1767us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.225833 (+ 2074us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.226899 (+ 1066us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.227143 (+ 244us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.228337 (+ 1194us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.228527 (+ 190us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":10991,"thread_start_us":11222,"threads_started":1}
I20251027 14:56:33.229604 21608 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.207208 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.3:45173 (local address 127.0.0.1:47890)
1027 14:56:33.222250 (+ 15042us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.222269 (+ 19us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:33.222394 (+ 125us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:33.223803 (+ 1409us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:33.223809 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:33.223839 (+ 30us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:33.224594 (+ 755us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.224607 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.226911 (+ 2304us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.226917 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:33.228839 (+ 1922us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.228854 (+ 15us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.229018 (+ 164us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.229043 (+ 25us) client_negotiation.cc:769] Sending connection context
1027 14:56:33.229110 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:56:33.229236 (+ 126us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14944,"thread_start_us":11616,"threads_started":1}
I20251027 14:56:33.231546 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.209797 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46086 (local address 127.0.104.1:34825)
1027 14:56:33.223233 (+ 13436us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.223244 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.223923 (+ 679us) server_negotiation.cc:408] Connection header received
1027 14:56:33.224030 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.224038 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.224223 (+ 185us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.224433 (+ 210us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:33.225448 (+ 1015us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.227182 (+ 1734us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.229689 (+ 2507us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.230788 (+ 1099us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.230989 (+ 201us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.231116 (+ 127us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.231337 (+ 221us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":13301,"thread_start_us":13189,"threads_started":1}
I20251027 14:56:33.231895 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.209258 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46086)
1027 14:56:33.223668 (+ 14410us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.223685 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:33.223800 (+ 115us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:33.224441 (+ 641us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:33.224446 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:33.224474 (+ 28us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:33.225258 (+ 784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.225271 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.227392 (+ 2121us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.227397 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:33.229181 (+ 1784us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.229193 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.229352 (+ 159us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.229375 (+ 23us) client_negotiation.cc:769] Sending connection context
1027 14:56:33.229434 (+ 59us) client_negotiation.cc:240] Negotiation successful
1027 14:56:33.229554 (+ 120us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":14317,"thread_start_us":14141,"threads_started":1}
I20251027 14:56:33.233084 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.207494 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47890 (local address 127.0.104.3:45173)
1027 14:56:33.222904 (+ 15410us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.222913 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.222948 (+ 35us) server_negotiation.cc:408] Connection header received
1027 14:56:33.223047 (+ 99us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.223055 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.223299 (+ 244us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.223507 (+ 208us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:33.224806 (+ 1299us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.226715 (+ 1909us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.229053 (+ 2338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.232450 (+ 3397us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.232594 (+ 144us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.232696 (+ 102us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.232832 (+ 136us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":15261,"thread_start_us":15118,"threads_started":1}
I20251027 14:56:33.233260 21401 tablet_service.cc:1505] Processing CreateTablet for tablet 2c76b88622824992ac1b8a4a6bbdc1cf (DEFAULT_TABLE table=test-table [id=2ed065a0652a4fbabe03b034032ac952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:33.234969 21401 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c76b88622824992ac1b8a4a6bbdc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:33.236295 21270 tablet_service.cc:1505] Processing CreateTablet for tablet 2c76b88622824992ac1b8a4a6bbdc1cf (DEFAULT_TABLE table=test-table [id=2ed065a0652a4fbabe03b034032ac952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:33.237555 21270 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c76b88622824992ac1b8a4a6bbdc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:33.237258 21531 tablet_service.cc:1505] Processing CreateTablet for tablet 2c76b88622824992ac1b8a4a6bbdc1cf (DEFAULT_TABLE table=test-table [id=2ed065a0652a4fbabe03b034032ac952]), partition=RANGE (key) PARTITION UNBOUNDED
I20251027 14:56:33.238634 21531 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2c76b88622824992ac1b8a4a6bbdc1cf. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:33.248143 21614 tablet_bootstrap.cc:492] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: Bootstrap starting.
I20251027 14:56:33.251358 21614 tablet_bootstrap.cc:654] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:33.252450 21614 log.cc:826] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:33.253295 21616 tablet_bootstrap.cc:492] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Bootstrap starting.
I20251027 14:56:33.254695 21615 tablet_bootstrap.cc:492] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Bootstrap starting.
I20251027 14:56:33.257813 21615 tablet_bootstrap.cc:654] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:33.257939 21616 tablet_bootstrap.cc:654] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:33.258520 21614 tablet_bootstrap.cc:492] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: No bootstrap required, opened a new log
I20251027 14:56:33.258805 21614 ts_tablet_manager.cc:1403] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20251027 14:56:33.258891 21615 log.cc:826] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:33.259158 21616 log.cc:826] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:33.262046 21615 tablet_bootstrap.cc:492] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: No bootstrap required, opened a new log
I20251027 14:56:33.262338 21615 ts_tablet_manager.cc:1403] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Time spent bootstrapping tablet: real 0.008s user 0.007s sys 0.000s
I20251027 14:56:33.263875 21616 tablet_bootstrap.cc:492] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: No bootstrap required, opened a new log
I20251027 14:56:33.264140 21616 ts_tablet_manager.cc:1403] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.000s
I20251027 14:56:33.265497 21614 raft_consensus.cc:359] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.265895 21614 raft_consensus.cc:385] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:33.266043 21614 raft_consensus.cc:740] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2244e513e4444c0a11b63b92eb7b068, State: Initialized, Role: FOLLOWER
I20251027 14:56:33.266728 21614 consensus_queue.cc:260] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.267829 21335 heartbeater.cc:499] Master 127.0.104.62:43083 was elected leader, sending a full tablet report...
I20251027 14:56:33.267913 21615 raft_consensus.cc:359] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.268298 21615 raft_consensus.cc:385] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:33.268422 21614 ts_tablet_manager.cc:1434] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: Time spent starting tablet: real 0.009s user 0.011s sys 0.000s
I20251027 14:56:33.268432 21615 raft_consensus.cc:740] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7b2ec6dece804303b489e4c0f9d73229, State: Initialized, Role: FOLLOWER
I20251027 14:56:33.269162 21615 consensus_queue.cc:260] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.270269 21466 heartbeater.cc:499] Master 127.0.104.62:43083 was elected leader, sending a full tablet report...
I20251027 14:56:33.270789 21615 ts_tablet_manager.cc:1434] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Time spent starting tablet: real 0.008s user 0.008s sys 0.000s
I20251027 14:56:33.270592 21616 raft_consensus.cc:359] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.270944 21616 raft_consensus.cc:385] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:33.271036 21616 raft_consensus.cc:740] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d7ab09d57db4de68dbd9333984c6848, State: Initialized, Role: FOLLOWER
I20251027 14:56:33.271647 21616 consensus_queue.cc:260] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.272696 21596 heartbeater.cc:499] Master 127.0.104.62:43083 was elected leader, sending a full tablet report...
I20251027 14:56:33.273885 21616 ts_tablet_manager.cc:1434] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Time spent starting tablet: real 0.010s user 0.004s sys 0.005s
W20251027 14:56:33.279335 21336 tablet.cc:2378] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:56:33.284725 21622 raft_consensus.cc:493] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20251027 14:56:33.284940 21622 raft_consensus.cc:515] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.286757 21622 leader_election.cc:290] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7b2ec6dece804303b489e4c0f9d73229 (127.0.104.2:39349), a2244e513e4444c0a11b63b92eb7b068 (127.0.104.1:34825)
I20251027 14:56:33.291260 21598 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.287015 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.2:39349 (local address 127.0.104.3:55505)
1027 14:56:33.287308 (+ 293us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.287328 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:33.287466 (+ 138us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:33.287768 (+ 302us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:33.287773 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:33.287809 (+ 36us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:33.288216 (+ 407us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.288230 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.289484 (+ 1254us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.289489 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:33.290560 (+ 1071us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.290571 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.290732 (+ 161us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.290777 (+ 45us) client_negotiation.cc:769] Sending connection context
1027 14:56:33.290844 (+ 67us) client_negotiation.cc:240] Negotiation successful
1027 14:56:33.290984 (+ 140us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":172}
I20251027 14:56:33.292006 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.287296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:55505 (local address 127.0.104.2:39349)
1027 14:56:33.287488 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.287494 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.287513 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:56:33.287584 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.287590 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.287663 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.287747 (+ 84us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:33.288370 (+ 623us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.289332 (+ 962us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.290698 (+ 1366us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.291654 (+ 956us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.291693 (+ 39us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.291755 (+ 62us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.291836 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":92}
I20251027 14:56:33.292943 21623 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.287058 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.104.3:56369)
1027 14:56:33.288757 (+ 1699us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.288774 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:56:33.288886 (+ 112us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:56:33.289144 (+ 258us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:56:33.289149 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:56:33.289180 (+ 31us) client_negotiation.cc:189] Negotiated authn=CERTIFICATE
1027 14:56:33.289723 (+ 543us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.289732 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.290905 (+ 1173us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.290909 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:56:33.292218 (+ 1309us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:56:33.292231 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.292434 (+ 203us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.292460 (+ 26us) client_negotiation.cc:769] Sending connection context
1027 14:56:33.292526 (+ 66us) client_negotiation.cc:240] Negotiation successful
1027 14:56:33.292641 (+ 115us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":1606,"thread_start_us":1439,"threads_started":1}
I20251027 14:56:33.292855 21421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c76b88622824992ac1b8a4a6bbdc1cf" candidate_uuid: "9d7ab09d57db4de68dbd9333984c6848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2ec6dece804303b489e4c0f9d73229" is_pre_election: true
I20251027 14:56:33.293309 21421 raft_consensus.cc:2468] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7ab09d57db4de68dbd9333984c6848 in term 0.
I20251027 14:56:33.293777 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.287356 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.104.3:56369 (local address 127.0.104.1:34825)
1027 14:56:33.287502 (+ 146us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.287508 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.288854 (+ 1346us) server_negotiation.cc:408] Connection header received
1027 14:56:33.288990 (+ 136us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.288994 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.289047 (+ 53us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.289120 (+ 73us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
1027 14:56:33.289863 (+ 743us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.290748 (+ 885us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.292347 (+ 1599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.293374 (+ 1027us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.293410 (+ 36us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.293487 (+ 77us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.293577 (+ 90us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:56:33.293937 21484 leader_election.cc:304] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b2ec6dece804303b489e4c0f9d73229, 9d7ab09d57db4de68dbd9333984c6848; no voters:
I20251027 14:56:33.294284 21622 raft_consensus.cc:2804] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20251027 14:56:33.294428 21622 raft_consensus.cc:493] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20251027 14:56:33.294534 21622 raft_consensus.cc:3060] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:33.294831 21290 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c76b88622824992ac1b8a4a6bbdc1cf" candidate_uuid: "9d7ab09d57db4de68dbd9333984c6848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2244e513e4444c0a11b63b92eb7b068" is_pre_election: true
I20251027 14:56:33.295295 21290 raft_consensus.cc:2468] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d7ab09d57db4de68dbd9333984c6848 in term 0.
I20251027 14:56:33.297247 21622 raft_consensus.cc:515] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.297804 21622 leader_election.cc:290] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [CANDIDATE]: Term 1 election: Requested vote from peers 7b2ec6dece804303b489e4c0f9d73229 (127.0.104.2:39349), a2244e513e4444c0a11b63b92eb7b068 (127.0.104.1:34825)
I20251027 14:56:33.298225 21421 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c76b88622824992ac1b8a4a6bbdc1cf" candidate_uuid: "9d7ab09d57db4de68dbd9333984c6848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7b2ec6dece804303b489e4c0f9d73229"
I20251027 14:56:33.298362 21290 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "2c76b88622824992ac1b8a4a6bbdc1cf" candidate_uuid: "9d7ab09d57db4de68dbd9333984c6848" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "a2244e513e4444c0a11b63b92eb7b068"
I20251027 14:56:33.298440 21421 raft_consensus.cc:3060] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:33.298533 21290 raft_consensus.cc:3060] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:33.301878 21421 raft_consensus.cc:2468] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7ab09d57db4de68dbd9333984c6848 in term 1.
I20251027 14:56:33.301878 21290 raft_consensus.cc:2468] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d7ab09d57db4de68dbd9333984c6848 in term 1.
I20251027 14:56:33.302309 21484 leader_election.cc:304] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7b2ec6dece804303b489e4c0f9d73229, 9d7ab09d57db4de68dbd9333984c6848; no voters:
I20251027 14:56:33.302649 21622 raft_consensus.cc:2804] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:33.303123 21622 raft_consensus.cc:697] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 1 LEADER]: Becoming Leader. State: Replica: 9d7ab09d57db4de68dbd9333984c6848, State: Running, Role: LEADER
I20251027 14:56:33.303659 21622 consensus_queue.cc:237] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } }
I20251027 14:56:33.308741 21148 catalog_manager.cc:5649] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 reported cstate change: term changed from 0 to 1, leader changed from <none> to 9d7ab09d57db4de68dbd9333984c6848 (127.0.104.3). New cstate: current_term: 1 leader_uuid: "9d7ab09d57db4de68dbd9333984c6848" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "9d7ab09d57db4de68dbd9333984c6848" member_type: VOTER last_known_addr { host: "127.0.104.3" port: 45173 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 } health_report { overall_health: UNKNOWN } } }
I20251027 14:56:33.334209 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.329290 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47902 (local address 127.0.104.3:45173)
1027 14:56:33.329501 (+ 211us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.329505 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.329524 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:56:33.329701 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.329706 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.329783 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.329905 (+ 122us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:56:33.330486 (+ 581us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.331384 (+ 898us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.332629 (+ 1245us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.332980 (+ 351us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.333136 (+ 156us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:56:33.333689 (+ 553us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:56:33.333799 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.333855 (+ 56us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.333944 (+ 89us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":102}
I20251027 14:56:33.341761 21421 raft_consensus.cc:1275] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 1 FOLLOWER]: Refusing update from remote peer 9d7ab09d57db4de68dbd9333984c6848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:33.341776 21290 raft_consensus.cc:1275] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 1 FOLLOWER]: Refusing update from remote peer 9d7ab09d57db4de68dbd9333984c6848: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:56:33.342695 21624 consensus_queue.cc:1048] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a2244e513e4444c0a11b63b92eb7b068" member_type: VOTER last_known_addr { host: "127.0.104.1" port: 34825 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:33.342919 21622 consensus_queue.cc:1048] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7b2ec6dece804303b489e4c0f9d73229" member_type: VOTER last_known_addr { host: "127.0.104.2" port: 39349 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:56:33.355372 21628 mvcc.cc:204] Tried to move back new op lower bound from 7215419364718325760 to 7215419364575956992. Current Snapshot: MvccSnapshot[applied={T|T < 7215419364718325760}]
I20251027 14:56:33.365356 21149 catalog_manager.cc:2507] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:47186:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:33.365553 21149 catalog_manager.cc:2755] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:47186:
table { table_name: "test-table" } modify_external_catalogs: true
I20251027 14:56:33.368711 21149 catalog_manager.cc:5936] T 00000000000000000000000000000000 P f2a972ff5003467da37e3d31e00a5fcc: Sending DeleteTablet for 3 replicas of tablet 2c76b88622824992ac1b8a4a6bbdc1cf
I20251027 14:56:33.369814 21531 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c76b88622824992ac1b8a4a6bbdc1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:33 UTC) from {username='slave'} at 127.0.0.1:47890
I20251027 14:56:33.369814 21401 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c76b88622824992ac1b8a4a6bbdc1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:33 UTC) from {username='slave'} at 127.0.0.1:48254
I20251027 14:56:33.370095 21270 tablet_service.cc:1552] Processing DeleteTablet for tablet 2c76b88622824992ac1b8a4a6bbdc1cf with delete_type TABLET_DATA_DELETED (Table deleted at 2025-10-27 14:56:33 UTC) from {username='slave'} at 127.0.0.1:46086
I20251027 14:56:33.371167 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21208
I20251027 14:56:33.382890 21636 tablet_replica.cc:333] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: stopping tablet replica
I20251027 14:56:33.383390 21636 raft_consensus.cc:2243] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:33.383905 21636 raft_consensus.cc:2272] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:33.389302 21636 ts_tablet_manager.cc:1916] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:33.390921 21637 tablet_replica.cc:333] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: stopping tablet replica
I20251027 14:56:33.391047 21638 tablet_replica.cc:333] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068: stopping tablet replica
I20251027 14:56:33.391111 21613 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.372460 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:46090 (local address 127.0.104.1:34825)
1027 14:56:33.372598 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.372604 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.382144 (+ 9540us) server_negotiation.cc:408] Connection header received
1027 14:56:33.382664 (+ 520us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.382672 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.382792 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.382870 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:33.384217 (+ 1347us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.385781 (+ 1564us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.387609 (+ 1828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.388149 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.389355 (+ 1206us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:33.389393 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:33.389412 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:33.390144 (+ 732us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:33.390210 (+ 66us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:33.390240 (+ 30us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:33.390322 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.390622 (+ 300us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.390744 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":30}
I20251027 14:56:33.391363 21638 raft_consensus.cc:2243] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 1 FOLLOWER]: Raft consensus shutting down.
I20251027 14:56:33.391360 21637 raft_consensus.cc:2243] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 1 LEADER]: Raft consensus shutting down.
I20251027 14:56:33.391839 21638 raft_consensus.cc:2272] T 2c76b88622824992ac1b8a4a6bbdc1cf P a2244e513e4444c0a11b63b92eb7b068 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:33.392025 21637 raft_consensus.cc:2272] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848 [term 1 FOLLOWER]: Raft consensus is shut down!
I20251027 14:56:33.392601 21330 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:33.394860 21637 ts_tablet_manager.cc:1916] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Deleting tablet data with delete state TABLET_DATA_DELETED
I20251027 14:56:33.397743 21636 ts_tablet_manager.cc:1929] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:33.397897 21636 log.cc:1199] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-1/wal/wals/2c76b88622824992ac1b8a4a6bbdc1cf
I20251027 14:56:33.398581 21636 ts_tablet_manager.cc:1950] T 2c76b88622824992ac1b8a4a6bbdc1cf P 7b2ec6dece804303b489e4c0f9d73229: Deleting consensus metadata
I20251027 14:56:33.400657 21133 catalog_manager.cc:4985] TS 7b2ec6dece804303b489e4c0f9d73229 (127.0.104.2:39349): tablet 2c76b88622824992ac1b8a4a6bbdc1cf (table test-table [id=2ed065a0652a4fbabe03b034032ac952]) successfully deleted
I20251027 14:56:33.403209 21637 ts_tablet_manager.cc:1929] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.2
I20251027 14:56:33.403376 21637 log.cc:1199] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Deleting WAL directory at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.EncryptionPolicyTest.TestEncryptionPolicy_5.1761576633729408-416-0/minicluster-data/ts-2/wal/wals/2c76b88622824992ac1b8a4a6bbdc1cf
I20251027 14:56:33.403873 21637 ts_tablet_manager.cc:1950] T 2c76b88622824992ac1b8a4a6bbdc1cf P 9d7ab09d57db4de68dbd9333984c6848: Deleting consensus metadata
I20251027 14:56:33.405652 21136 catalog_manager.cc:4985] TS 9d7ab09d57db4de68dbd9333984c6848 (127.0.104.3:45173): tablet 2c76b88622824992ac1b8a4a6bbdc1cf (table test-table [id=2ed065a0652a4fbabe03b034032ac952]) successfully deleted
I20251027 14:56:33.487668 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21208
I20251027 14:56:33.504148 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21339
W20251027 14:56:33.504309 21135 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv got EOF from 127.0.104.1:34825 (error 108)
I20251027 14:56:33.506541 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.506070 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46106)
1027 14:56:33.506259 (+ 189us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.506374 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":66}
W20251027 14:56:33.506884 21135 catalog_manager.cc:4712] TS a2244e513e4444c0a11b63b92eb7b068 (127.0.104.1:34825): DeleteTablet:TABLET_DATA_DELETED RPC failed for tablet 2c76b88622824992ac1b8a4a6bbdc1cf: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
I20251027 14:56:33.510587 21610 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.505055 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48268 (local address 127.0.104.2:39349)
1027 14:56:33.505231 (+ 176us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.505237 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.505258 (+ 21us) server_negotiation.cc:408] Connection header received
1027 14:56:33.505330 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.505336 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.505450 (+ 114us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.505561 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:33.506223 (+ 662us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.507276 (+ 1053us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.508440 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.508829 (+ 389us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.509277 (+ 448us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:33.509309 (+ 32us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:33.509322 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:33.509910 (+ 588us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:33.509956 (+ 46us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:33.509974 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:33.510072 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.510191 (+ 119us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.510298 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:56:33.511256 21461 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:33.560490 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.559917 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46118)
1027 14:56:33.560130 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.560242 (+ 112us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20251027 14:56:33.561582 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.561215 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46132)
1027 14:56:33.561363 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.561452 (+ 89us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251027 14:56:33.598238 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21339
I20251027 14:56:33.619447 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21469
I20251027 14:56:33.621999 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.621587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46144)
1027 14:56:33.621749 (+ 162us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.621826 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":54}
I20251027 14:56:33.623062 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.622739 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46146)
1027 14:56:33.622870 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.622949 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
I20251027 14:56:33.627204 21611 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.620185 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47916 (local address 127.0.104.3:45173)
1027 14:56:33.620410 (+ 225us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:33.620421 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:33.620441 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:56:33.620630 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:33.620635 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:33.620721 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:33.620806 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:33.621546 (+ 740us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.623066 (+ 1520us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:33.624757 (+ 1691us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:33.625198 (+ 441us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:33.625756 (+ 558us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:33.625790 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:33.625803 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:33.626447 (+ 644us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:33.626518 (+ 71us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:33.626539 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:33.626625 (+ 86us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:33.626780 (+ 155us) server_negotiation.cc:300] Negotiation successful
1027 14:56:33.626872 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":79}
I20251027 14:56:33.627926 21591 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:33.703455 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.702977 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46160)
1027 14:56:33.703186 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.703293 (+ 107us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":75}
I20251027 14:56:33.704674 21612 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:33.704129 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.1:34825 (local address 127.0.0.1:46176)
1027 14:56:33.704348 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1027 14:56:33.704448 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.1:34825: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":109}
I20251027 14:56:33.744966 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21469
I20251027 14:56:33.760804 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21118
I20251027 14:56:33.761835 21179 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:33.869396 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21118
2025-10-27T14:56:33Z chronyd exiting
[ OK ] EncryptionPolicyTest.TestEncryptionPolicy/5 (2657 ms)
[----------] 6 tests from EncryptionPolicyTest (24464 ms total)
[----------] 18 tests from AuthTokenIssuingTest
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/0
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:33 dist-test-slave-hxxt krb5kdc[21647](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:33 dist-test-slave-hxxt krb5kdc[21647](info): set up 2 sockets
Oct 27 14:56:33 dist-test-slave-hxxt krb5kdc[21647](info): commencing operation
krb5kdc: starting...
W20251027 14:56:35.752717 416 mini_kdc.cc:121] Time spent starting KDC: real 1.844s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:35 dist-test-slave-hxxt krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:35Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:35Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:36.031781 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43313
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43313
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:36.356585 21663 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:36.356910 21663 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:36.357050 21663 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:36.365156 21663 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:36.365267 21663 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:36.365334 21663 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:36.365401 21663 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:36.365475 21663 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:36.376012 21663 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43313
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:36.377590 21663 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:36.378896 21663 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:36.387218 21669 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:36.387269 21671 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:36.387723 21668 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:36.388161 21663 server_base.cc:1047] running on GCE node
I20251027 14:56:36.388983 21663 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:36.390390 21663 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:36.391615 21663 hybrid_clock.cc:648] HybridClock initialized: now 1761576996391548 us; error 40 us; skew 500 ppm
I20251027 14:56:36.391939 21663 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:36 dist-test-slave-hxxt krb5kdc[21647](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576996, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:36.395022 21663 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:56:36.396555 21663 webserver.cc:492] Webserver started at http://127.0.0.1:33939/ using document root <none> and password file <none>
I20251027 14:56:36.397269 21663 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:36.397348 21663 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:36.397665 21663 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:36.400254 21663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "a2ed90da9555477e915b2b4c0d63b27a"
format_stamp: "Formatted at 2025-10-27 14:56:36 on dist-test-slave-hxxt"
server_key: "7c22708bfcdc9b7319e634271126c70f"
server_key_iv: "64c0ec993f4d5c436662b076b852b766"
server_key_version: "encryptionkey@0"
I20251027 14:56:36.401068 21663 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "a2ed90da9555477e915b2b4c0d63b27a"
format_stamp: "Formatted at 2025-10-27 14:56:36 on dist-test-slave-hxxt"
server_key: "7c22708bfcdc9b7319e634271126c70f"
server_key_iv: "64c0ec993f4d5c436662b076b852b766"
server_key_version: "encryptionkey@0"
I20251027 14:56:36.407785 21663 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.002s
I20251027 14:56:36.412494 21678 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:36.414069 21663 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:36.414235 21663 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "a2ed90da9555477e915b2b4c0d63b27a"
format_stamp: "Formatted at 2025-10-27 14:56:36 on dist-test-slave-hxxt"
server_key: "7c22708bfcdc9b7319e634271126c70f"
server_key_iv: "64c0ec993f4d5c436662b076b852b766"
server_key_version: "encryptionkey@0"
I20251027 14:56:36.414460 21663 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:36.430843 21663 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:36.439051 21663 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:36.439379 21663 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:36.458652 21663 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43313
I20251027 14:56:36.458670 21729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:43313 every 8 connection(s)
I20251027 14:56:36.460109 21663 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:36.464553 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21663
I20251027 14:56:36.464567 21730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:36.464752 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_0.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:36.465302 416 external_mini_cluster.cc:1468] Setting key 56085aa1d6f6b15933cc1e0d3b0ced25
I20251027 14:56:36.477654 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a: Bootstrap starting.
Oct 27 14:56:36 dist-test-slave-hxxt krb5kdc[21647](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576995, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251027 14:56:36.481340 21730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:36.482717 21730 log.cc:826] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:36.486337 21730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a: No bootstrap required, opened a new log
I20251027 14:56:36.487988 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:36.467218 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:58045 (local address 127.0.0.1:43313)
1027 14:56:36.467959 (+ 741us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:36.467973 (+ 14us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:36.468022 (+ 49us) server_negotiation.cc:408] Connection header received
1027 14:56:36.470088 (+ 2066us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:36.470118 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:36.470678 (+ 560us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:36.471246 (+ 568us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:36.472789 (+ 1543us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:36.474750 (+ 1961us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:36.475834 (+ 1084us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:36.476417 (+ 583us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:36.479654 (+ 3237us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:36.479772 (+ 118us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:36.479793 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:36.479837 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:36.483513 (+ 3676us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:36.484500 (+ 987us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:36.484509 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:36.484518 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:36.484643 (+ 125us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:36.485009 (+ 366us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:36.485015 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:36.485018 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:36.485619 (+ 601us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:36.485959 (+ 340us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:36.486278 (+ 319us) server_negotiation.cc:300] Negotiation successful
1027 14:56:36.486619 (+ 341us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":308,"threads_started":1}
I20251027 14:56:36.492295 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:56:36.493089 21730 raft_consensus.cc:359] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed90da9555477e915b2b4c0d63b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20251027 14:56:36.493474 21730 raft_consensus.cc:385] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:36.493573 21730 raft_consensus.cc:740] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a2ed90da9555477e915b2b4c0d63b27a, State: Initialized, Role: FOLLOWER
I20251027 14:56:36.494244 21730 consensus_queue.cc:260] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed90da9555477e915b2b4c0d63b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20251027 14:56:36.494469 21730 raft_consensus.cc:399] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:36.494599 21730 raft_consensus.cc:493] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:36.494762 21730 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:36.497936 21730 raft_consensus.cc:515] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed90da9555477e915b2b4c0d63b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20251027 14:56:36.498553 21730 leader_election.cc:304] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: a2ed90da9555477e915b2b4c0d63b27a; no voters:
I20251027 14:56:36.499229 21730 leader_election.cc:290] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:36.499317 21740 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:36.500627 21740 raft_consensus.cc:697] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [term 1 LEADER]: Becoming Leader. State: Replica: a2ed90da9555477e915b2b4c0d63b27a, State: Running, Role: LEADER
I20251027 14:56:36.501149 21740 consensus_queue.cc:237] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed90da9555477e915b2b4c0d63b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } }
I20251027 14:56:36.501529 21730 sys_catalog.cc:565] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:36.503626 21742 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [sys.catalog]: SysCatalogTable state changed. Reason: New leader a2ed90da9555477e915b2b4c0d63b27a. Latest consensus state: current_term: 1 leader_uuid: "a2ed90da9555477e915b2b4c0d63b27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed90da9555477e915b2b4c0d63b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } } }
I20251027 14:56:36.503912 21742 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:36.504635 21741 sys_catalog.cc:455] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "a2ed90da9555477e915b2b4c0d63b27a" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "a2ed90da9555477e915b2b4c0d63b27a" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43313 } } }
I20251027 14:56:36.504801 21741 sys_catalog.cc:458] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:36.505587 21747 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:36.514556 21733 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:36.497628 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48791 (local address 127.0.0.1:43313)
1027 14:56:36.497825 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:36.497832 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:36.498289 (+ 457us) server_negotiation.cc:408] Connection header received
1027 14:56:36.498482 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:36.498486 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:36.498551 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:36.498629 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:36.500903 (+ 2274us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:36.502196 (+ 1293us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:36.503765 (+ 1569us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:36.504243 (+ 478us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:36.506583 (+ 2340us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:36.506621 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:36.506627 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:56:36.506668 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:36.510492 (+ 3824us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:36.511497 (+ 1005us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:36.511504 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:36.511508 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:36.511601 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:56:36.512084 (+ 483us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:56:36.512090 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:56:36.512094 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:56:36.512411 (+ 317us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:36.512704 (+ 293us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:36.513962 (+ 1258us) server_negotiation.cc:300] Negotiation successful
1027 14:56:36.514165 (+ 203us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:56:36.519503 21747 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:36.529584 21747 catalog_manager.cc:1357] Generated new cluster ID: 61b3cbc64871446abc913ca50da21111
I20251027 14:56:36.529704 21747 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:36.550242 21747 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:36.551983 21747 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:36.563218 21747 catalog_manager.cc:6022] T 00000000000000000000000000000000 P a2ed90da9555477e915b2b4c0d63b27a: Generated new TSK 0
I20251027 14:56:36.564029 21747 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:56:36.570307 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21663
I20251027 14:56:36.571839 21725 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:56:36.659848 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21663
W20251027 14:56:36.674167 21645 connection.cc:537] client connection to 127.0.0.1:43313 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43313): Connection reset by peer (error 104)
2025-10-27T14:56:36Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/0 (2785 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/1
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:36 dist-test-slave-hxxt krb5kdc[21762](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:36 dist-test-slave-hxxt krb5kdc[21762](info): set up 2 sockets
Oct 27 14:56:36 dist-test-slave-hxxt krb5kdc[21762](info): commencing operation
krb5kdc: starting...
W20251027 14:56:38.540714 416 mini_kdc.cc:121] Time spent starting KDC: real 1.847s user 0.003s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:38 dist-test-slave-hxxt krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576998, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:38.841784 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39151
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:42441
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:39151
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:39.164956 21778 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:39.165261 21778 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:39.165391 21778 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:39.173661 21778 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:39.173764 21778 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:39.173830 21778 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:39.173895 21778 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:39.173978 21778 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:39.184788 21778 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:42441
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:39151
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:39151
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:39.186383 21778 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:39.187724 21778 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:39.195808 21784 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:39.195777 21783 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:39.196471 21786 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:39.196970 21778 server_base.cc:1047] running on GCE node
I20251027 14:56:39.197743 21778 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:39.199168 21778 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:39.200423 21778 hybrid_clock.cc:648] HybridClock initialized: now 1761576999200322 us; error 51 us; skew 500 ppm
I20251027 14:56:39.200764 21778 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:39 dist-test-slave-hxxt krb5kdc[21762](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761576999, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:39.203811 21778 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:56:39.205484 21778 webserver.cc:492] Webserver started at http://127.0.0.1:45683/ using document root <none> and password file <none>
I20251027 14:56:39.206231 21778 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:39.206342 21778 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:39.206698 21778 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:39.209443 21778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "0e5e4ec50e674773b9d68870765fde34"
format_stamp: "Formatted at 2025-10-27 14:56:39 on dist-test-slave-hxxt"
server_key: "bf5fb2b1e02736ea92040c359d4f5a30"
server_key_iv: "dc5746468fe4510c036af0da1602f25d"
server_key_version: "encryptionkey@0"
I20251027 14:56:39.210201 21778 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "0e5e4ec50e674773b9d68870765fde34"
format_stamp: "Formatted at 2025-10-27 14:56:39 on dist-test-slave-hxxt"
server_key: "bf5fb2b1e02736ea92040c359d4f5a30"
server_key_iv: "dc5746468fe4510c036af0da1602f25d"
server_key_version: "encryptionkey@0"
I20251027 14:56:39.215852 21778 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:56:39.219533 21793 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:39.221026 21778 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20251027 14:56:39.221200 21778 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "0e5e4ec50e674773b9d68870765fde34"
format_stamp: "Formatted at 2025-10-27 14:56:39 on dist-test-slave-hxxt"
server_key: "bf5fb2b1e02736ea92040c359d4f5a30"
server_key_iv: "dc5746468fe4510c036af0da1602f25d"
server_key_version: "encryptionkey@0"
I20251027 14:56:39.221434 21778 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:39.240314 21778 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:39.244212 21778 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:39.244501 21778 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:39.263655 21778 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:39151
I20251027 14:56:39.263674 21844 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:39151 every 8 connection(s)
I20251027 14:56:39.265190 21778 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:39.269433 21845 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:39.275465 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21778
I20251027 14:56:39.275696 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_1.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:39.276160 416 external_mini_cluster.cc:1468] Setting key 9575989bca0d1cc0b82e261fb765701a
I20251027 14:56:39.280894 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34: Bootstrap starting.
I20251027 14:56:39.284049 21845 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:39.285305 21845 log.cc:826] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:39.287998 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:39.278003 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38453 (local address 127.0.0.1:39151)
1027 14:56:39.278834 (+ 831us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:39.278847 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:39.278897 (+ 50us) server_negotiation.cc:408] Connection header received
1027 14:56:39.280053 (+ 1156us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:39.280075 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:39.280574 (+ 499us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:39.280988 (+ 414us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:39.281777 (+ 789us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:39.283151 (+ 1374us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:39.284327 (+ 1176us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:39.284891 (+ 564us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:39.285411 (+ 520us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:39.285493 (+ 82us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:39.285516 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:39.286279 (+ 763us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:39.286355 (+ 76us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:39.286395 (+ 40us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:39.286590 (+ 195us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:39.286885 (+ 295us) server_negotiation.cc:300] Negotiation successful
1027 14:56:39.287079 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":447,"threads_started":1}
I20251027 14:56:39.288597 21845 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34: No bootstrap required, opened a new log
I20251027 14:56:39.291714 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:56:39.295042 21845 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5e4ec50e674773b9d68870765fde34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39151 } }
I20251027 14:56:39.295506 21845 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:39.295624 21845 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0e5e4ec50e674773b9d68870765fde34, State: Initialized, Role: FOLLOWER
I20251027 14:56:39.296363 21845 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5e4ec50e674773b9d68870765fde34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39151 } }
I20251027 14:56:39.296635 21845 raft_consensus.cc:399] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:39.296772 21845 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:39.296947 21845 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:39.299892 21845 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5e4ec50e674773b9d68870765fde34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39151 } }
I20251027 14:56:39.300555 21845 leader_election.cc:304] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 0e5e4ec50e674773b9d68870765fde34; no voters:
I20251027 14:56:39.301190 21845 leader_election.cc:290] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:39.301275 21855 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:39.301915 21855 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [term 1 LEADER]: Becoming Leader. State: Replica: 0e5e4ec50e674773b9d68870765fde34, State: Running, Role: LEADER
I20251027 14:56:39.302410 21855 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5e4ec50e674773b9d68870765fde34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39151 } }
I20251027 14:56:39.303876 21848 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:39.297115 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:34445 (local address 127.0.0.1:39151)
1027 14:56:39.297305 (+ 190us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:39.297311 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:39.297677 (+ 366us) server_negotiation.cc:408] Connection header received
1027 14:56:39.297844 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:39.297850 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:39.297935 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:39.298044 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:39.298962 (+ 918us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:39.299775 (+ 813us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:56:39.301365 (+ 1590us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:56:39.302022 (+ 657us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:56:39.302657 (+ 635us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:39.302720 (+ 63us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:39.302727 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:39.302750 (+ 23us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:39.302808 (+ 58us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:39.302830 (+ 22us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:39.302944 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:39.303483 (+ 539us) server_negotiation.cc:300] Negotiation successful
1027 14:56:39.303547 (+ 64us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:56:39.304762 21845 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:39.305655 21857 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0e5e4ec50e674773b9d68870765fde34. Latest consensus state: current_term: 1 leader_uuid: "0e5e4ec50e674773b9d68870765fde34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5e4ec50e674773b9d68870765fde34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39151 } } }
I20251027 14:56:39.305935 21857 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:39.306201 21856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "0e5e4ec50e674773b9d68870765fde34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "0e5e4ec50e674773b9d68870765fde34" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 39151 } } }
I20251027 14:56:39.306483 21856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:39.310101 21860 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:39.314447 21860 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:39.322659 21860 catalog_manager.cc:1357] Generated new cluster ID: 78c3b345487c42baaa6a12335b293d0c
I20251027 14:56:39.322769 21860 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:39.348457 21860 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:39.349707 21860 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:39.357295 21860 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 0e5e4ec50e674773b9d68870765fde34: Generated new TSK 0
I20251027 14:56:39.358369 21860 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:56:39.374104 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21778
W20251027 14:56:39.375914 21840 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:38453
W20251027 14:56:39.376899 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21778: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:56:39.377070 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21778
W20251027 14:56:39.391433 21760 connection.cc:537] client connection to 127.0.0.1:39151 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:39151): Connection reset by peer (error 104)
2025-10-27T14:56:39Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/1 (2717 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/2
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:39 dist-test-slave-hxxt krb5kdc[21876](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:39 dist-test-slave-hxxt krb5kdc[21876](info): set up 2 sockets
Oct 27 14:56:39 dist-test-slave-hxxt krb5kdc[21876](info): commencing operation
krb5kdc: starting...
W20251027 14:56:41.252657 416 mini_kdc.cc:121] Time spent starting KDC: real 1.842s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:41 dist-test-slave-hxxt krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577001, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:41Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:41.536521 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35563
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36041
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:41.863618 21892 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:41.863988 21892 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:41.864130 21892 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:41.872458 21892 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:41.872591 21892 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:41.872695 21892 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:41.872759 21892 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:41.872840 21892 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:41.883729 21892 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35563
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36041
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:41.885390 21892 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:41.886802 21892 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:41.895254 21897 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:41.895460 21900 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:41.895251 21898 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:41.896536 21892 server_base.cc:1047] running on GCE node
I20251027 14:56:41.897394 21892 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:41.898880 21892 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:41.900115 21892 hybrid_clock.cc:648] HybridClock initialized: now 1761577001900054 us; error 39 us; skew 500 ppm
I20251027 14:56:41.900501 21892 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:56:41 dist-test-slave-hxxt krb5kdc[21876](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577001, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:56:41.903626 21892 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:56:41.905385 21892 webserver.cc:492] Webserver started at http://127.0.0.1:40467/ using document root <none> and password file <none>
I20251027 14:56:41.906136 21892 fs_manager.cc:362] Metadata directory not provided
I20251027 14:56:41.906229 21892 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:56:41.906580 21892 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:56:41.909217 21892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "4b2b341709ff4925aa105145f78c93e0"
format_stamp: "Formatted at 2025-10-27 14:56:41 on dist-test-slave-hxxt"
server_key: "f115b9ce099554e218cd02c820133906"
server_key_iv: "318482f46f002ccff424ba320a46fa95"
server_key_version: "encryptionkey@0"
I20251027 14:56:41.909983 21892 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "4b2b341709ff4925aa105145f78c93e0"
format_stamp: "Formatted at 2025-10-27 14:56:41 on dist-test-slave-hxxt"
server_key: "f115b9ce099554e218cd02c820133906"
server_key_iv: "318482f46f002ccff424ba320a46fa95"
server_key_version: "encryptionkey@0"
I20251027 14:56:41.915369 21892 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:56:41.919616 21907 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:56:41.921110 21892 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:56:41.921269 21892 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "4b2b341709ff4925aa105145f78c93e0"
format_stamp: "Formatted at 2025-10-27 14:56:41 on dist-test-slave-hxxt"
server_key: "f115b9ce099554e218cd02c820133906"
server_key_iv: "318482f46f002ccff424ba320a46fa95"
server_key_version: "encryptionkey@0"
I20251027 14:56:41.921486 21892 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:56:41.927683 21892 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:56:41.935110 21892 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:56:41.935365 21892 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:56:41.954380 21892 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36041
I20251027 14:56:41.954403 21958 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36041 every 8 connection(s)
I20251027 14:56:41.955899 21892 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:56:41.959646 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 21892
I20251027 14:56:41.959842 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_2.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:56:41.960162 21959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:56:41.960215 416 external_mini_cluster.cc:1468] Setting key db3f93e423bf7ec832e728e20a39132c
I20251027 14:56:41.968430 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:41.962002 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41551 (local address 127.0.0.1:36041)
1027 14:56:41.962777 (+ 775us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:41.962793 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:41.962847 (+ 54us) server_negotiation.cc:408] Connection header received
1027 14:56:41.964408 (+ 1561us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:41.964438 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:41.964932 (+ 494us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:41.965436 (+ 504us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:41.966543 (+ 1107us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:41.966552 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:41.966569 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:41.966583 (+ 14us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:41.966655 (+ 72us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:41.966678 (+ 23us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:41.966898 (+ 220us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:41.967054 (+ 156us) server_negotiation.cc:300] Negotiation successful
1027 14:56:41.967247 (+ 193us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":561,"thread_start_us":355,"threads_started":1}
I20251027 14:56:41.972311 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:56:41.973723 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0: Bootstrap starting.
I20251027 14:56:41.977217 21959 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0: Neither blocks nor log segments found. Creating new log.
I20251027 14:56:41.978344 21959 log.cc:826] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0: Log is configured to *not* fsync() on all Append() calls
I20251027 14:56:41.978906 21962 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:56:41.976498 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45969 (local address 127.0.0.1:36041)
1027 14:56:41.976639 (+ 141us) server_negotiation.cc:207] Beginning negotiation
1027 14:56:41.976643 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:56:41.977101 (+ 458us) server_negotiation.cc:408] Connection header received
1027 14:56:41.977239 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:56:41.977245 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:56:41.977312 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:56:41.977416 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:56:41.978273 (+ 857us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:56:41.978279 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:56:41.978283 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:56:41.978298 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:56:41.978353 (+ 55us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:56:41.978374 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:56:41.978487 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1027 14:56:41.978571 (+ 84us) server_negotiation.cc:300] Negotiation successful
1027 14:56:41.978624 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":44}
I20251027 14:56:41.981549 21959 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0: No bootstrap required, opened a new log
I20251027 14:56:41.986811 21959 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b341709ff4925aa105145f78c93e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20251027 14:56:41.987140 21959 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:56:41.987244 21959 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4b2b341709ff4925aa105145f78c93e0, State: Initialized, Role: FOLLOWER
I20251027 14:56:41.987799 21959 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b341709ff4925aa105145f78c93e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20251027 14:56:41.987984 21959 raft_consensus.cc:399] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:56:41.988102 21959 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:56:41.988237 21959 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:56:41.990486 21959 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b341709ff4925aa105145f78c93e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20251027 14:56:41.990967 21959 leader_election.cc:304] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 4b2b341709ff4925aa105145f78c93e0; no voters:
I20251027 14:56:41.991497 21959 leader_election.cc:290] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:56:41.991647 21969 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:56:41.992638 21969 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [term 1 LEADER]: Becoming Leader. State: Replica: 4b2b341709ff4925aa105145f78c93e0, State: Running, Role: LEADER
I20251027 14:56:41.993108 21969 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b341709ff4925aa105145f78c93e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } }
I20251027 14:56:41.993551 21959 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:56:41.995764 21971 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4b2b341709ff4925aa105145f78c93e0. Latest consensus state: current_term: 1 leader_uuid: "4b2b341709ff4925aa105145f78c93e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b341709ff4925aa105145f78c93e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20251027 14:56:41.995834 21970 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "4b2b341709ff4925aa105145f78c93e0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "4b2b341709ff4925aa105145f78c93e0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36041 } } }
I20251027 14:56:41.996043 21971 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:41.996124 21970 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0 [sys.catalog]: This master's current role is: LEADER
I20251027 14:56:41.996930 21978 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:56:42.002621 21978 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:56:42.012287 21978 catalog_manager.cc:1357] Generated new cluster ID: 6bdbf4829a5940e595f55b38bbf79335
I20251027 14:56:42.012459 21978 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:56:42.044365 21978 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:56:42.044809 21978 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:56:42.058064 21978 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 4b2b341709ff4925aa105145f78c93e0: Generated new TSK 0
I20251027 14:56:42.058951 21978 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:56:42.075976 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21892
W20251027 14:56:42.077376 21954 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:41551
W20251027 14:56:42.078138 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 21892: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:56:42.078259 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 21892
2025-10-27T14:56:42Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/2 (2700 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/3
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:56:42 dist-test-slave-hxxt krb5kdc[21990](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:56:42 dist-test-slave-hxxt krb5kdc[21990](info): set up 2 sockets
Oct 27 14:56:42 dist-test-slave-hxxt krb5kdc[21990](info): commencing operation
krb5kdc: starting...
W20251027 14:56:43.976593 416 mini_kdc.cc:121] Time spent starting KDC: real 1.865s user 0.003s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:56:44 dist-test-slave-hxxt krb5kdc[21990](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577004, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:56:44Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:56:44Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:56:44.261122 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:34853
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43327
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:56:44.585366 22006 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:56:44.585661 22006 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:56:44.585760 22006 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:56:44.593806 22006 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:56:44.593907 22006 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:56:44.593966 22006 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:56:44.594012 22006 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:56:44.594067 22006 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:56:44.604621 22006 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:34853
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43327
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:56:44.606178 22006 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:56:44.607457 22006 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:56:44.615013 22011 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:56:44.617965 22012 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:44.622411 22006 server_base.cc:1047] running on GCE node
W20251027 14:56:44.623221 22014 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:56:44.624357 22006 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:56:44.626001 22006 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:56:44.627225 22006 hybrid_clock.cc:648] HybridClock initialized: now 1761577004627149 us; error 52 us; skew 500 ppm
I20251027 14:56:44.627559 22006 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20251027 14:57:02.645447 22006 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
Runtime error: kinit failed: unable to login from keytab: Cannot contact any KDC for realm 'KRBTEST.COM'
/home/jenkins-slave/workspace/build_and_test_flaky@2/src/kudu/integration-tests/security-itest.cc:1479: Failure
Failed
Bad status: Runtime error: failed to start masters: Unable to start Master at index 0: /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu: process exited with non-zero status 1
2025-10-27T14:57:02Z chronyd exiting
I20251027 14:57:02.750350 416 test_util.cc:183] -----------------------------------------------
I20251027 14:57:02.750478 416 test_util.cc:184] Had failures, leaving test files at /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_3.1761576633729408-416-0
[ FAILED ] AuthTokenIssuingTest.ChannelConfidentiality/3, where GetParam() = 80-byte object <02-00 00-00 BE-BE BE-BE E8-24 01-00 70-60 00-00 08-00 00-00 00-00 00-00 72-65 71-75 69-72 65-64 00-BE BE-BE BE-BE BE-BE 08-25 01-00 70-60 00-00 08-00 00-00 00-00 00-00 72-65 71-75 69-72 65-64 00-BE BE-BE BE-BE BE-BE 00-00 01-BE BE-BE BE-BE> (20643 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/4
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:02 dist-test-slave-hxxt krb5kdc[22020](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:02 dist-test-slave-hxxt krb5kdc[22020](info): set up 2 sockets
Oct 27 14:57:02 dist-test-slave-hxxt krb5kdc[22020](info): commencing operation
krb5kdc: starting...
W20251027 14:57:04.612636 416 mini_kdc.cc:121] Time spent starting KDC: real 1.858s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:04 dist-test-slave-hxxt krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577024, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:04Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:04.900758 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37861
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43675
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:37861
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:05.225664 22036 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:05.225984 22036 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:05.226087 22036 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:05.234273 22036 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:05.234385 22036 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:05.234441 22036 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:05.234490 22036 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:05.234551 22036 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:05.245028 22036 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43675
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:37861
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:37861
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:05.246570 22036 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:05.247852 22036 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:05.255642 22042 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:05.255915 22044 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:05.255664 22041 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:05.256706 22036 server_base.cc:1047] running on GCE node
I20251027 14:57:05.257436 22036 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:05.258806 22036 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:05.260032 22036 hybrid_clock.cc:648] HybridClock initialized: now 1761577025259965 us; error 46 us; skew 500 ppm
I20251027 14:57:05.260411 22036 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:05 dist-test-slave-hxxt krb5kdc[22020](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577025, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:05.263340 22036 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:05.264827 22036 webserver.cc:492] Webserver started at http://127.0.0.1:42359/ using document root <none> and password file <none>
I20251027 14:57:05.265523 22036 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:05.265599 22036 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:05.265885 22036 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:05.268683 22036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79"
format_stamp: "Formatted at 2025-10-27 14:57:05 on dist-test-slave-hxxt"
server_key: "5fe589b4be7cd1fbb42556ccb3fdb82d"
server_key_iv: "b6d8720b250faf0a6e058eb47b34439a"
server_key_version: "encryptionkey@0"
I20251027 14:57:05.269486 22036 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79"
format_stamp: "Formatted at 2025-10-27 14:57:05 on dist-test-slave-hxxt"
server_key: "5fe589b4be7cd1fbb42556ccb3fdb82d"
server_key_iv: "b6d8720b250faf0a6e058eb47b34439a"
server_key_version: "encryptionkey@0"
I20251027 14:57:05.275127 22036 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20251027 14:57:05.278574 22051 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:05.279939 22036 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:57:05.280090 22036 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79"
format_stamp: "Formatted at 2025-10-27 14:57:05 on dist-test-slave-hxxt"
server_key: "5fe589b4be7cd1fbb42556ccb3fdb82d"
server_key_iv: "b6d8720b250faf0a6e058eb47b34439a"
server_key_version: "encryptionkey@0"
I20251027 14:57:05.280283 22036 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:05.297272 22036 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:05.305289 22036 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:05.305526 22036 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:05.323951 22036 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:37861
I20251027 14:57:05.323974 22102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:37861 every 8 connection(s)
I20251027 14:57:05.325448 22036 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:05.329582 22103 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:05.334481 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22036
I20251027 14:57:05.334679 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_4.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:05.335158 416 external_mini_cluster.cc:1468] Setting key 75cfa39e9456fbd19e0f7ce699d79207
I20251027 14:57:05.340775 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79: Bootstrap starting.
I20251027 14:57:05.343910 22103 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:05.345170 22103 log.cc:826] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:05.346992 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:05.337095 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:47105 (local address 127.0.0.1:37861)
1027 14:57:05.337871 (+ 776us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:05.337884 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:05.337931 (+ 47us) server_negotiation.cc:408] Connection header received
1027 14:57:05.339107 (+ 1176us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:05.339131 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:05.339551 (+ 420us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:05.339976 (+ 425us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:05.340856 (+ 880us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:05.342220 (+ 1364us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:05.343387 (+ 1167us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:05.343882 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:05.344441 (+ 559us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:05.344507 (+ 66us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:05.344522 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:05.345044 (+ 522us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:05.345091 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:05.345113 (+ 22us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:05.345204 (+ 91us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:05.345480 (+ 276us) server_negotiation.cc:300] Negotiation successful
1027 14:57:05.345736 (+ 256us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":575,"thread_start_us":334,"threads_started":1}
I20251027 14:57:05.348441 22103 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79: No bootstrap required, opened a new log
I20251027 14:57:05.350601 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:05.353950 22103 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20251027 14:57:05.354279 22103 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:05.354385 22103 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e1ad06cdbb346bfb476c8bb8fda5f79, State: Initialized, Role: FOLLOWER
I20251027 14:57:05.355118 22103 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20251027 14:57:05.355317 22103 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:05.355443 22103 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:05.355599 22103 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:05.357949 22103 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20251027 14:57:05.358458 22103 leader_election.cc:304] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e1ad06cdbb346bfb476c8bb8fda5f79; no voters:
I20251027 14:57:05.359381 22103 leader_election.cc:290] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:05.359483 22113 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:05.360167 22106 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:05.354671 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35939 (local address 127.0.0.1:37861)
1027 14:57:05.354910 (+ 239us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:05.354916 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:05.355212 (+ 296us) server_negotiation.cc:408] Connection header received
1027 14:57:05.355378 (+ 166us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:05.355384 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:05.355459 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:05.355537 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:05.356276 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:05.357188 (+ 912us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:05.358404 (+ 1216us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:05.358951 (+ 547us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:05.359343 (+ 392us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:05.359397 (+ 54us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:05.359402 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:05.359417 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:05.359458 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:05.359484 (+ 26us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:05.359561 (+ 77us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:05.359762 (+ 201us) server_negotiation.cc:300] Negotiation successful
1027 14:57:05.359867 (+ 105us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":126}
I20251027 14:57:05.360260 22113 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [term 1 LEADER]: Becoming Leader. State: Replica: 9e1ad06cdbb346bfb476c8bb8fda5f79, State: Running, Role: LEADER
I20251027 14:57:05.360981 22113 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } }
I20251027 14:57:05.362674 22103 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:05.364570 22115 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } } }
I20251027 14:57:05.364663 22114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e1ad06cdbb346bfb476c8bb8fda5f79. Latest consensus state: current_term: 1 leader_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e1ad06cdbb346bfb476c8bb8fda5f79" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 37861 } } }
I20251027 14:57:05.364876 22115 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:05.364955 22114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:05.366482 22120 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:05.372131 22120 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:05.382541 22120 catalog_manager.cc:1357] Generated new cluster ID: 206463d067454dad88f5cd48ac7f0292
I20251027 14:57:05.382678 22120 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:05.391425 22120 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:05.393190 22120 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:05.417328 22120 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9e1ad06cdbb346bfb476c8bb8fda5f79: Generated new TSK 0
I20251027 14:57:05.418463 22120 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:05.428869 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22036
W20251027 14:57:05.429965 22098 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:47105
W20251027 14:57:05.430951 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22036: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:05.431067 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22036
2025-10-27T14:57:05Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/4 (2709 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/5
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:05 dist-test-slave-hxxt krb5kdc[22134](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:05 dist-test-slave-hxxt krb5kdc[22134](info): set up 2 sockets
Oct 27 14:57:05 dist-test-slave-hxxt krb5kdc[22134](info): commencing operation
krb5kdc: starting...
W20251027 14:57:07.316751 416 mini_kdc.cc:121] Time spent starting KDC: real 1.853s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:07 dist-test-slave-hxxt krb5kdc[22134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577027, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:07Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:07.600580 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45983
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:43003
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:45983
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:07.929234 22150 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:07.929548 22150 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:07.929682 22150 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:07.937830 22150 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:07.937935 22150 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:07.937999 22150 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:07.938066 22150 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:07.938141 22150 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:07.948697 22150 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:43003
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:45983
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:45983
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:07.950207 22150 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:07.951503 22150 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:07.958930 22155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:07.958873 22156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:07.959415 22158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:07.960068 22150 server_base.cc:1047] running on GCE node
I20251027 14:57:07.960847 22150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:07.962117 22150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:07.963317 22150 hybrid_clock.cc:648] HybridClock initialized: now 1761577027963254 us; error 38 us; skew 500 ppm
I20251027 14:57:07.963625 22150 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:07 dist-test-slave-hxxt krb5kdc[22134](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577027, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:07.966517 22150 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:07.967978 22150 webserver.cc:492] Webserver started at http://127.0.0.1:33729/ using document root <none> and password file <none>
I20251027 14:57:07.968690 22150 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:07.968768 22150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:07.969059 22150 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:07.971472 22150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "6268378462fa45b49f54fdc8b74fdf0e"
format_stamp: "Formatted at 2025-10-27 14:57:07 on dist-test-slave-hxxt"
server_key: "b089b2c8b4de319f71aa289f41c6184a"
server_key_iv: "7d001a0923f8761920188c0862b0663a"
server_key_version: "encryptionkey@0"
I20251027 14:57:07.972159 22150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "6268378462fa45b49f54fdc8b74fdf0e"
format_stamp: "Formatted at 2025-10-27 14:57:07 on dist-test-slave-hxxt"
server_key: "b089b2c8b4de319f71aa289f41c6184a"
server_key_iv: "7d001a0923f8761920188c0862b0663a"
server_key_version: "encryptionkey@0"
I20251027 14:57:07.977214 22150 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20251027 14:57:07.980322 22165 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:07.981566 22150 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20251027 14:57:07.981711 22150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "6268378462fa45b49f54fdc8b74fdf0e"
format_stamp: "Formatted at 2025-10-27 14:57:07 on dist-test-slave-hxxt"
server_key: "b089b2c8b4de319f71aa289f41c6184a"
server_key_iv: "7d001a0923f8761920188c0862b0663a"
server_key_version: "encryptionkey@0"
I20251027 14:57:07.981900 22150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:07.987958 22150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:07.991837 22150 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:07.992069 22150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:08.012113 22150 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:45983
I20251027 14:57:08.012154 22216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:45983 every 8 connection(s)
I20251027 14:57:08.013679 22150 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:08.017794 22217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:08.023810 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22150
I20251027 14:57:08.024014 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_5.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:08.024510 416 external_mini_cluster.cc:1468] Setting key 9aa398e29ef41bb55b8002b56bec3260
I20251027 14:57:08.028478 22217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e: Bootstrap starting.
I20251027 14:57:08.031701 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:08.026278 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49865 (local address 127.0.0.1:45983)
1027 14:57:08.027033 (+ 755us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:08.027045 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:08.027089 (+ 44us) server_negotiation.cc:408] Connection header received
1027 14:57:08.028201 (+ 1112us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:08.028223 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:08.028650 (+ 427us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:08.029007 (+ 357us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:08.030134 (+ 1127us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:08.030142 (+ 8us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:08.030154 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:08.030169 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:08.030231 (+ 62us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:08.030261 (+ 30us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:08.030489 (+ 228us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:08.030620 (+ 131us) server_negotiation.cc:300] Negotiation successful
1027 14:57:08.030817 (+ 197us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":573,"thread_start_us":346,"threads_started":1}
I20251027 14:57:08.031942 22217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:08.033046 22217 log.cc:826] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:08.035535 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:08.036247 22217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e: No bootstrap required, opened a new log
I20251027 14:57:08.042906 22220 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:08.040533 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48035 (local address 127.0.0.1:45983)
1027 14:57:08.040706 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:08.040712 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:08.041073 (+ 361us) server_negotiation.cc:408] Connection header received
1027 14:57:08.041199 (+ 126us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:08.041205 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:08.041259 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:08.041357 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:08.042186 (+ 829us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:08.042193 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:08.042198 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:08.042213 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:08.042261 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:08.042279 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:08.042389 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:08.042576 (+ 187us) server_negotiation.cc:300] Negotiation successful
1027 14:57:08.042637 (+ 61us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":56}
I20251027 14:57:08.042953 22217 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6268378462fa45b49f54fdc8b74fdf0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20251027 14:57:08.043332 22217 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:08.043455 22217 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6268378462fa45b49f54fdc8b74fdf0e, State: Initialized, Role: FOLLOWER
I20251027 14:57:08.044126 22217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6268378462fa45b49f54fdc8b74fdf0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20251027 14:57:08.044396 22217 raft_consensus.cc:399] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:08.044531 22217 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:08.044662 22217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:08.046759 22217 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6268378462fa45b49f54fdc8b74fdf0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20251027 14:57:08.047209 22217 leader_election.cc:304] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 6268378462fa45b49f54fdc8b74fdf0e; no voters:
I20251027 14:57:08.047709 22217 leader_election.cc:290] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:08.047832 22227 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:08.048478 22227 raft_consensus.cc:697] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [term 1 LEADER]: Becoming Leader. State: Replica: 6268378462fa45b49f54fdc8b74fdf0e, State: Running, Role: LEADER
I20251027 14:57:08.048925 22227 consensus_queue.cc:237] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6268378462fa45b49f54fdc8b74fdf0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } }
I20251027 14:57:08.049543 22217 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:08.051684 22229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 6268378462fa45b49f54fdc8b74fdf0e. Latest consensus state: current_term: 1 leader_uuid: "6268378462fa45b49f54fdc8b74fdf0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6268378462fa45b49f54fdc8b74fdf0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } } }
I20251027 14:57:08.051721 22228 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "6268378462fa45b49f54fdc8b74fdf0e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "6268378462fa45b49f54fdc8b74fdf0e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 45983 } } }
I20251027 14:57:08.052049 22228 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:08.052044 22229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:08.053535 22232 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:08.058734 22232 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:08.068220 22232 catalog_manager.cc:1357] Generated new cluster ID: 3023093d499b48f3819ab6386a9491c7
I20251027 14:57:08.068395 22232 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:08.083510 22232 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:08.083983 22232 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:08.095033 22232 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 6268378462fa45b49f54fdc8b74fdf0e: Generated new TSK 0
I20251027 14:57:08.096158 22232 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:08.104784 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22150
W20251027 14:57:08.105809 22212 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49865
W20251027 14:57:08.106498 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22150: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:08.106648 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22150
2025-10-27T14:57:08Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/5 (2679 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/6
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:08 dist-test-slave-hxxt krb5kdc[22248](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:08 dist-test-slave-hxxt krb5kdc[22248](info): set up 2 sockets
Oct 27 14:57:08 dist-test-slave-hxxt krb5kdc[22248](info): commencing operation
krb5kdc: starting...
W20251027 14:57:09.996706 416 mini_kdc.cc:121] Time spent starting KDC: real 1.853s user 0.004s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:10 dist-test-slave-hxxt krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:10Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:10Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:10.281518 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43289
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:39779
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:43289
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:10.611308 22264 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:10.611639 22264 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:10.611779 22264 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:10.619966 22264 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:10.620090 22264 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:10.620183 22264 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:10.620245 22264 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:10.620324 22264 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:10.631130 22264 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:39779
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:43289
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:43289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:10.632830 22264 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:10.634227 22264 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:10.642292 22270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:10.642725 22272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:10.642880 22269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:10.643473 22264 server_base.cc:1047] running on GCE node
I20251027 14:57:10.644297 22264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:10.645754 22264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:10.646986 22264 hybrid_clock.cc:648] HybridClock initialized: now 1761577030646923 us; error 45 us; skew 500 ppm
I20251027 14:57:10.647367 22264 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:10 dist-test-slave-hxxt krb5kdc[22248](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577030, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:10.650457 22264 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:10.652019 22264 webserver.cc:492] Webserver started at http://127.0.0.1:34259/ using document root <none> and password file <none>
I20251027 14:57:10.652952 22264 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:10.653055 22264 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:10.653431 22264 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:10.656078 22264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "214e41b0d7c24c2ca73ac97646444c9e"
format_stamp: "Formatted at 2025-10-27 14:57:10 on dist-test-slave-hxxt"
server_key: "94aa50ac8e1aa8e2d141fcfbda31e76d"
server_key_iv: "b0b409c5911b7e576ae7d3b2eb6de6b6"
server_key_version: "encryptionkey@0"
I20251027 14:57:10.656934 22264 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "214e41b0d7c24c2ca73ac97646444c9e"
format_stamp: "Formatted at 2025-10-27 14:57:10 on dist-test-slave-hxxt"
server_key: "94aa50ac8e1aa8e2d141fcfbda31e76d"
server_key_iv: "b0b409c5911b7e576ae7d3b2eb6de6b6"
server_key_version: "encryptionkey@0"
I20251027 14:57:10.662390 22264 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:57:10.665760 22279 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:10.667042 22264 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20251027 14:57:10.667213 22264 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "214e41b0d7c24c2ca73ac97646444c9e"
format_stamp: "Formatted at 2025-10-27 14:57:10 on dist-test-slave-hxxt"
server_key: "94aa50ac8e1aa8e2d141fcfbda31e76d"
server_key_iv: "b0b409c5911b7e576ae7d3b2eb6de6b6"
server_key_version: "encryptionkey@0"
I20251027 14:57:10.667420 22264 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:10.680609 22264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:10.687566 22264 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:10.687812 22264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:10.707001 22264 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:43289
I20251027 14:57:10.707039 22330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:43289 every 8 connection(s)
I20251027 14:57:10.708541 22264 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:10.712769 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:10.715274 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22264
I20251027 14:57:10.715497 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_6.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:10.716009 416 external_mini_cluster.cc:1468] Setting key be807a86a43082c8fb6bd6d1f01bcd47
I20251027 14:57:10.725411 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e: Bootstrap starting.
Oct 27 14:57:10 dist-test-slave-hxxt krb5kdc[22248](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577030, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251027 14:57:10.728826 22331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:10.730116 22331 log.cc:826] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:10.733567 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e: No bootstrap required, opened a new log
I20251027 14:57:10.737807 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:10.717998 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:45491 (local address 127.0.0.1:43289)
1027 14:57:10.718764 (+ 766us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:10.718780 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:10.718827 (+ 47us) server_negotiation.cc:408] Connection header received
1027 14:57:10.720397 (+ 1570us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:10.720437 (+ 40us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:10.720966 (+ 529us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:10.721503 (+ 537us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:10.722965 (+ 1462us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:10.724902 (+ 1937us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:10.726013 (+ 1111us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:10.726531 (+ 518us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:10.730023 (+ 3492us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:10.730089 (+ 66us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:10.730107 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:10.730141 (+ 34us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:10.733630 (+ 3489us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:10.734346 (+ 716us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:10.734355 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:10.734363 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:10.734492 (+ 129us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:10.734844 (+ 352us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:10.734849 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:10.734852 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:10.735452 (+ 600us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:10.735785 (+ 333us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:10.736081 (+ 296us) server_negotiation.cc:300] Negotiation successful
1027 14:57:10.736482 (+ 401us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":548,"thread_start_us":340,"threads_started":1}
I20251027 14:57:10.740464 22331 raft_consensus.cc:359] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e41b0d7c24c2ca73ac97646444c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43289 } }
I20251027 14:57:10.740803 22331 raft_consensus.cc:385] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:10.740892 22331 raft_consensus.cc:740] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 214e41b0d7c24c2ca73ac97646444c9e, State: Initialized, Role: FOLLOWER
I20251027 14:57:10.741503 22331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e41b0d7c24c2ca73ac97646444c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43289 } }
I20251027 14:57:10.741734 22331 raft_consensus.cc:399] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:10.741770 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:10.741859 22331 raft_consensus.cc:493] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:10.741995 22331 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:10.744153 22331 raft_consensus.cc:515] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e41b0d7c24c2ca73ac97646444c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43289 } }
I20251027 14:57:10.744791 22331 leader_election.cc:304] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 214e41b0d7c24c2ca73ac97646444c9e; no voters:
I20251027 14:57:10.745414 22331 leader_election.cc:290] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:10.745527 22340 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:10.746248 22340 raft_consensus.cc:697] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [term 1 LEADER]: Becoming Leader. State: Replica: 214e41b0d7c24c2ca73ac97646444c9e, State: Running, Role: LEADER
I20251027 14:57:10.746866 22340 consensus_queue.cc:237] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e41b0d7c24c2ca73ac97646444c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43289 } }
I20251027 14:57:10.750046 22343 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "214e41b0d7c24c2ca73ac97646444c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e41b0d7c24c2ca73ac97646444c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43289 } } }
I20251027 14:57:10.750429 22343 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:10.751427 22341 sys_catalog.cc:455] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 214e41b0d7c24c2ca73ac97646444c9e. Latest consensus state: current_term: 1 leader_uuid: "214e41b0d7c24c2ca73ac97646444c9e" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "214e41b0d7c24c2ca73ac97646444c9e" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 43289 } } }
I20251027 14:57:10.751732 22341 sys_catalog.cc:458] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:10.753053 22331 sys_catalog.cc:565] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:10.763345 22345 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:10.766049 22334 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:10.745755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:57513 (local address 127.0.0.1:43289)
1027 14:57:10.745918 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:10.745923 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:10.748881 (+ 2958us) server_negotiation.cc:408] Connection header received
1027 14:57:10.749141 (+ 260us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:10.749147 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:10.749231 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:10.749307 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:10.751631 (+ 2324us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:10.753009 (+ 1378us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:10.754778 (+ 1769us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:10.755280 (+ 502us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:10.758381 (+ 3101us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:10.758419 (+ 38us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:10.758434 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:10.758474 (+ 40us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:10.762409 (+ 3935us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:10.763363 (+ 954us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:10.763370 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:10.763373 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:10.763489 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:10.764000 (+ 511us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:10.764007 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:10.764010 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:10.764397 (+ 387us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:10.764723 (+ 326us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:10.765592 (+ 869us) server_negotiation.cc:300] Negotiation successful
1027 14:57:10.765788 (+ 196us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":46}
I20251027 14:57:10.769207 22345 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:10.778813 22345 catalog_manager.cc:1357] Generated new cluster ID: d7c34e4080d74ad2807f034978c40497
I20251027 14:57:10.778929 22345 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:10.805096 22345 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:10.806372 22345 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:10.820744 22345 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 214e41b0d7c24c2ca73ac97646444c9e: Generated new TSK 0
I20251027 14:57:10.821633 22345 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:10.838443 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22264
I20251027 14:57:10.840677 22326 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:10.932967 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22264
W20251027 14:57:10.947439 22246 connection.cc:537] client connection to 127.0.0.1:43289 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:43289): Connection reset by peer (error 104)
2025-10-27T14:57:10Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/6 (2823 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/7
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:11 dist-test-slave-hxxt krb5kdc[22364](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:11 dist-test-slave-hxxt krb5kdc[22364](info): set up 2 sockets
Oct 27 14:57:11 dist-test-slave-hxxt krb5kdc[22364](info): commencing operation
krb5kdc: starting...
W20251027 14:57:12.812649 416 mini_kdc.cc:121] Time spent starting KDC: real 1.846s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:12 dist-test-slave-hxxt krb5kdc[22364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577032, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:12Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:12Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:13.095309 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34557
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:45865
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:34557
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:13.414433 22380 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:13.414741 22380 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:13.414872 22380 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:13.422912 22380 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:13.423019 22380 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:13.423089 22380 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:13.423154 22380 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:13.423235 22380 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:13.433640 22380 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:45865
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:34557
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:34557
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:13.435180 22380 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:13.436537 22380 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:13.444393 22386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:13.444401 22385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:13.445017 22388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:13.445333 22380 server_base.cc:1047] running on GCE node
I20251027 14:57:13.446059 22380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:13.447423 22380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:13.448632 22380 hybrid_clock.cc:648] HybridClock initialized: now 1761577033448572 us; error 39 us; skew 500 ppm
I20251027 14:57:13.448938 22380 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:13 dist-test-slave-hxxt krb5kdc[22364](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577033, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:13.451869 22380 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:13.453480 22380 webserver.cc:492] Webserver started at http://127.0.0.1:38649/ using document root <none> and password file <none>
I20251027 14:57:13.454214 22380 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:13.454301 22380 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:13.454648 22380 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:13.457289 22380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "65c151b2f5024d0c9831ac5201aadab0"
format_stamp: "Formatted at 2025-10-27 14:57:13 on dist-test-slave-hxxt"
server_key: "c397be9d73f8ebf60af8ea300065f7b2"
server_key_iv: "a796f1021ed902dda6e405be609a47e1"
server_key_version: "encryptionkey@0"
I20251027 14:57:13.458039 22380 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "65c151b2f5024d0c9831ac5201aadab0"
format_stamp: "Formatted at 2025-10-27 14:57:13 on dist-test-slave-hxxt"
server_key: "c397be9d73f8ebf60af8ea300065f7b2"
server_key_iv: "a796f1021ed902dda6e405be609a47e1"
server_key_version: "encryptionkey@0"
I20251027 14:57:13.463336 22380 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20251027 14:57:13.466770 22395 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:13.468060 22380 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:57:13.468225 22380 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "65c151b2f5024d0c9831ac5201aadab0"
format_stamp: "Formatted at 2025-10-27 14:57:13 on dist-test-slave-hxxt"
server_key: "c397be9d73f8ebf60af8ea300065f7b2"
server_key_iv: "a796f1021ed902dda6e405be609a47e1"
server_key_version: "encryptionkey@0"
I20251027 14:57:13.468452 22380 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:13.493530 22380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:13.497171 22380 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:13.497427 22380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:13.516083 22380 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:34557
I20251027 14:57:13.516103 22446 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:34557 every 8 connection(s)
I20251027 14:57:13.517561 22380 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:13.521606 22447 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:13.527719 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22380
I20251027 14:57:13.527920 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_7.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:13.528467 416 external_mini_cluster.cc:1468] Setting key e9bd94b759d2c1dc20d2c01a2a4fdd98
I20251027 14:57:13.532449 22447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0: Bootstrap starting.
I20251027 14:57:13.535981 22447 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:13.537345 22447 log.cc:826] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:13.540458 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:13.530262 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:47445 (local address 127.0.0.1:34557)
1027 14:57:13.531004 (+ 742us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:13.531015 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:13.531054 (+ 39us) server_negotiation.cc:408] Connection header received
1027 14:57:13.532202 (+ 1148us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:13.532226 (+ 24us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:13.532696 (+ 470us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:13.533049 (+ 353us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:13.533942 (+ 893us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:13.535426 (+ 1484us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:13.536964 (+ 1538us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:13.537439 (+ 475us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:13.538051 (+ 612us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:13.538125 (+ 74us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:13.538146 (+ 21us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:13.538851 (+ 705us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:13.538896 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:13.538919 (+ 23us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:13.539053 (+ 134us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:13.539321 (+ 268us) server_negotiation.cc:300] Negotiation successful
1027 14:57:13.539503 (+ 182us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":350,"threads_started":1}
I20251027 14:57:13.540513 22447 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0: No bootstrap required, opened a new log
I20251027 14:57:13.544065 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:13.546624 22447 raft_consensus.cc:359] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c151b2f5024d0c9831ac5201aadab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20251027 14:57:13.547075 22447 raft_consensus.cc:385] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:13.547187 22447 raft_consensus.cc:740] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 65c151b2f5024d0c9831ac5201aadab0, State: Initialized, Role: FOLLOWER
I20251027 14:57:13.547959 22447 consensus_queue.cc:260] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c151b2f5024d0c9831ac5201aadab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20251027 14:57:13.548216 22447 raft_consensus.cc:399] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:13.548353 22447 raft_consensus.cc:493] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:13.548542 22447 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:13.551280 22447 raft_consensus.cc:515] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c151b2f5024d0c9831ac5201aadab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20251027 14:57:13.551959 22447 leader_election.cc:304] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 65c151b2f5024d0c9831ac5201aadab0; no voters:
I20251027 14:57:13.552594 22447 leader_election.cc:290] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:13.552729 22457 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:13.553305 22457 raft_consensus.cc:697] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [term 1 LEADER]: Becoming Leader. State: Replica: 65c151b2f5024d0c9831ac5201aadab0, State: Running, Role: LEADER
I20251027 14:57:13.553941 22457 consensus_queue.cc:237] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c151b2f5024d0c9831ac5201aadab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } }
I20251027 14:57:13.555650 22450 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:13.549258 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:40451 (local address 127.0.0.1:34557)
1027 14:57:13.549440 (+ 182us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:13.549447 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:13.549821 (+ 374us) server_negotiation.cc:408] Connection header received
1027 14:57:13.549999 (+ 178us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:13.550006 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:13.550085 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:13.550196 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:13.550860 (+ 664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:13.552134 (+ 1274us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:13.553751 (+ 1617us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:13.554378 (+ 627us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:13.554768 (+ 390us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:13.554828 (+ 60us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:13.554835 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:13.554857 (+ 22us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:13.554898 (+ 41us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:13.554918 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:13.555012 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:13.555208 (+ 196us) server_negotiation.cc:300] Negotiation successful
1027 14:57:13.555264 (+ 56us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:57:13.556720 22457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 65c151b2f5024d0c9831ac5201aadab0. Latest consensus state: current_term: 1 leader_uuid: "65c151b2f5024d0c9831ac5201aadab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c151b2f5024d0c9831ac5201aadab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } } }
I20251027 14:57:13.557080 22457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:13.558065 22459 sys_catalog.cc:455] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "65c151b2f5024d0c9831ac5201aadab0" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "65c151b2f5024d0c9831ac5201aadab0" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 34557 } } }
I20251027 14:57:13.558390 22459 sys_catalog.cc:458] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:13.560442 22447 sys_catalog.cc:565] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:13.560626 22461 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:13.577841 22461 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:13.587602 22461 catalog_manager.cc:1357] Generated new cluster ID: f537676539434ef29b5c7b597939cb2d
I20251027 14:57:13.587738 22461 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:13.603082 22461 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:13.604486 22461 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:13.612628 22461 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 65c151b2f5024d0c9831ac5201aadab0: Generated new TSK 0
I20251027 14:57:13.613476 22461 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:13.628599 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22380
W20251027 14:57:13.629760 22442 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.25:47445
W20251027 14:57:13.630522 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22380: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:13.630643 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22380
W20251027 14:57:13.644901 22362 connection.cc:537] client connection to 127.0.0.1:34557 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:34557): Connection reset by peer (error 104)
2025-10-27T14:57:13Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/7 (2697 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/8
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:13 dist-test-slave-hxxt krb5kdc[22478](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:13 dist-test-slave-hxxt krb5kdc[22478](info): set up 2 sockets
Oct 27 14:57:13 dist-test-slave-hxxt krb5kdc[22478](info): commencing operation
krb5kdc: starting...
W20251027 14:57:15.508904 416 mini_kdc.cc:121] Time spent starting KDC: real 1.845s user 0.007s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:15 dist-test-slave-hxxt krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577035, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:15.792506 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38615
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:41061
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38615
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:16.116760 22494 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:16.117095 22494 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:16.117233 22494 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:16.125743 22494 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:16.125859 22494 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:16.125928 22494 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:16.125996 22494 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:16.126070 22494 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:16.136896 22494 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:41061
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38615
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:16.138489 22494 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:16.140069 22494 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:16.148780 22499 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:16.148802 22500 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:16.149308 22502 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:16.149737 22494 server_base.cc:1047] running on GCE node
I20251027 14:57:16.150490 22494 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:16.151926 22494 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:16.153159 22494 hybrid_clock.cc:648] HybridClock initialized: now 1761577036153092 us; error 38 us; skew 500 ppm
I20251027 14:57:16.153494 22494 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:16 dist-test-slave-hxxt krb5kdc[22478](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577036, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:16.156644 22494 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:16.158318 22494 webserver.cc:492] Webserver started at http://127.0.0.1:44019/ using document root <none> and password file <none>
I20251027 14:57:16.159080 22494 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:16.159171 22494 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:16.159528 22494 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:16.162180 22494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "f8aa52dfeecf4c1ba8a761600319ab6b"
format_stamp: "Formatted at 2025-10-27 14:57:16 on dist-test-slave-hxxt"
server_key: "32e60fa53114e5275b895180ee7383b1"
server_key_iv: "937f9634ac4bb5f8d6ace8805cc4c4ec"
server_key_version: "encryptionkey@0"
I20251027 14:57:16.162940 22494 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "f8aa52dfeecf4c1ba8a761600319ab6b"
format_stamp: "Formatted at 2025-10-27 14:57:16 on dist-test-slave-hxxt"
server_key: "32e60fa53114e5275b895180ee7383b1"
server_key_iv: "937f9634ac4bb5f8d6ace8805cc4c4ec"
server_key_version: "encryptionkey@0"
I20251027 14:57:16.168201 22494 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20251027 14:57:16.171618 22509 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:16.172977 22494 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:57:16.173125 22494 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "f8aa52dfeecf4c1ba8a761600319ab6b"
format_stamp: "Formatted at 2025-10-27 14:57:16 on dist-test-slave-hxxt"
server_key: "32e60fa53114e5275b895180ee7383b1"
server_key_iv: "937f9634ac4bb5f8d6ace8805cc4c4ec"
server_key_version: "encryptionkey@0"
I20251027 14:57:16.173319 22494 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:16.179289 22494 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:16.187160 22494 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:16.187417 22494 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:16.205968 22494 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38615
I20251027 14:57:16.205992 22560 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:38615 every 8 connection(s)
I20251027 14:57:16.207451 22494 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:16.211745 22561 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:16.215363 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22494
I20251027 14:57:16.215520 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_8.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:16.215953 416 external_mini_cluster.cc:1468] Setting key 18cc258f1b3ecf0d71a37baac459a99b
I20251027 14:57:16.223399 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:16.217775 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:44969 (local address 127.0.0.1:38615)
1027 14:57:16.218569 (+ 794us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:16.218581 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:16.218632 (+ 51us) server_negotiation.cc:408] Connection header received
1027 14:57:16.219785 (+ 1153us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:16.219808 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:16.220281 (+ 473us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:16.220797 (+ 516us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:16.221807 (+ 1010us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:16.221814 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:16.221827 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:16.221838 (+ 11us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:16.221887 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:16.221905 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:16.222079 (+ 174us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:16.222208 (+ 129us) server_negotiation.cc:300] Negotiation successful
1027 14:57:16.222433 (+ 225us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":592,"thread_start_us":370,"threads_started":1}
I20251027 14:57:16.223837 22561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b: Bootstrap starting.
I20251027 14:57:16.227463 22561 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:16.227608 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:16.228519 22561 log.cc:826] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:16.231617 22561 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b: No bootstrap required, opened a new log
I20251027 14:57:16.234807 22564 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:16.232167 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:48015 (local address 127.0.0.1:38615)
1027 14:57:16.232298 (+ 131us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:16.232302 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:16.232785 (+ 483us) server_negotiation.cc:408] Connection header received
1027 14:57:16.232973 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:16.232978 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:16.233025 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:16.233106 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:16.233982 (+ 876us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:16.233988 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:16.233994 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:16.234009 (+ 15us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:16.234053 (+ 44us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:16.234068 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:16.234172 (+ 104us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:16.234359 (+ 187us) server_negotiation.cc:300] Negotiation successful
1027 14:57:16.234425 (+ 66us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":39}
I20251027 14:57:16.237337 22561 raft_consensus.cc:359] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38615 } }
I20251027 14:57:16.237658 22561 raft_consensus.cc:385] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:16.237761 22561 raft_consensus.cc:740] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8aa52dfeecf4c1ba8a761600319ab6b, State: Initialized, Role: FOLLOWER
I20251027 14:57:16.238364 22561 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38615 } }
I20251027 14:57:16.238600 22561 raft_consensus.cc:399] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:16.238740 22561 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:16.238909 22561 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:16.241168 22561 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38615 } }
I20251027 14:57:16.241623 22561 leader_election.cc:304] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f8aa52dfeecf4c1ba8a761600319ab6b; no voters:
I20251027 14:57:16.242170 22561 leader_election.cc:290] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:16.242295 22571 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:16.242909 22571 raft_consensus.cc:697] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [term 1 LEADER]: Becoming Leader. State: Replica: f8aa52dfeecf4c1ba8a761600319ab6b, State: Running, Role: LEADER
I20251027 14:57:16.243342 22571 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38615 } }
I20251027 14:57:16.243997 22561 sys_catalog.cc:565] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:16.245986 22573 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8aa52dfeecf4c1ba8a761600319ab6b. Latest consensus state: current_term: 1 leader_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38615 } } }
I20251027 14:57:16.246290 22573 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:16.246598 22572 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f8aa52dfeecf4c1ba8a761600319ab6b" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38615 } } }
I20251027 14:57:16.246884 22572 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:16.247354 22579 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:16.253463 22579 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:16.267061 22579 catalog_manager.cc:1357] Generated new cluster ID: a3d0b231869a440483414300308d47cd
I20251027 14:57:16.267196 22579 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:16.297955 22579 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:16.298327 22579 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:16.305774 22579 catalog_manager.cc:6022] T 00000000000000000000000000000000 P f8aa52dfeecf4c1ba8a761600319ab6b: Generated new TSK 0
I20251027 14:57:16.306579 22579 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:16.323133 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22494
W20251027 14:57:16.324735 22556 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.25:44969
W20251027 14:57:16.325448 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22494: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:16.325604 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22494
2025-10-27T14:57:16Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/8 (2694 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/9
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:16 dist-test-slave-hxxt krb5kdc[22592](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:16 dist-test-slave-hxxt krb5kdc[22592](info): set up 2 sockets
Oct 27 14:57:16 dist-test-slave-hxxt krb5kdc[22592](info): commencing operation
krb5kdc: starting...
W20251027 14:57:18.224692 416 mini_kdc.cc:121] Time spent starting KDC: real 1.866s user 0.010s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:18 dist-test-slave-hxxt krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:18Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:18Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:18.512984 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36607
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:35747
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:36607
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:18.836941 22608 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:18.837255 22608 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:18.837399 22608 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:18.847987 22608 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:18.848126 22608 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:18.848220 22608 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:18.848289 22608 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:18.848369 22608 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:18.863956 22608 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:35747
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:36607
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:36607
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:18.866385 22608 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:18.868244 22608 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:18.877179 22613 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:18.878782 22616 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:18.878751 22614 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:18.886097 22608 server_base.cc:1047] running on GCE node
I20251027 14:57:18.886902 22608 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:18.888410 22608 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:18.889632 22608 hybrid_clock.cc:648] HybridClock initialized: now 1761577038889562 us; error 44 us; skew 500 ppm
I20251027 14:57:18.889954 22608 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:18 dist-test-slave-hxxt krb5kdc[22592](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577038, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:18.893018 22608 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:18.894511 22608 webserver.cc:492] Webserver started at http://127.0.0.1:38237/ using document root <none> and password file <none>
I20251027 14:57:18.895247 22608 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:18.895328 22608 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:18.895704 22608 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:18.898308 22608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "3814c3b95c3842aca80003a5a7457abe"
format_stamp: "Formatted at 2025-10-27 14:57:18 on dist-test-slave-hxxt"
server_key: "cf61e4614ead6402a569188ef9f13fce"
server_key_iv: "988a009d2828fbeadd78d0a88739ab2d"
server_key_version: "encryptionkey@0"
I20251027 14:57:18.899098 22608 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "3814c3b95c3842aca80003a5a7457abe"
format_stamp: "Formatted at 2025-10-27 14:57:18 on dist-test-slave-hxxt"
server_key: "cf61e4614ead6402a569188ef9f13fce"
server_key_iv: "988a009d2828fbeadd78d0a88739ab2d"
server_key_version: "encryptionkey@0"
I20251027 14:57:18.904517 22608 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20251027 14:57:18.907985 22623 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:18.909324 22608 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:57:18.909518 22608 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "3814c3b95c3842aca80003a5a7457abe"
format_stamp: "Formatted at 2025-10-27 14:57:18 on dist-test-slave-hxxt"
server_key: "cf61e4614ead6402a569188ef9f13fce"
server_key_iv: "988a009d2828fbeadd78d0a88739ab2d"
server_key_version: "encryptionkey@0"
I20251027 14:57:18.909718 22608 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:18.935993 22608 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:18.940037 22608 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:18.940276 22608 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:18.958364 22608 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:36607
I20251027 14:57:18.958372 22674 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:36607 every 8 connection(s)
I20251027 14:57:18.959842 22608 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:18.963883 22675 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:18.965972 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22608
I20251027 14:57:18.966171 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_9.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:18.966648 416 external_mini_cluster.cc:1468] Setting key e54bce4b64874e288f4332a4d3db15e4
I20251027 14:57:18.975672 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe: Bootstrap starting.
Oct 27 14:57:18 dist-test-slave-hxxt krb5kdc[22592](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577038, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.0.1@KRBTEST.COM
I20251027 14:57:18.979549 22675 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:18.980852 22675 log.cc:826] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:18.983997 22675 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe: No bootstrap required, opened a new log
I20251027 14:57:18.987843 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:18.968368 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:60711 (local address 127.0.0.1:36607)
1027 14:57:18.969178 (+ 810us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:18.969193 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:18.969235 (+ 42us) server_negotiation.cc:408] Connection header received
1027 14:57:18.970414 (+ 1179us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:18.970435 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:18.970819 (+ 384us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:18.971157 (+ 338us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:18.973198 (+ 2041us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:18.974599 (+ 1401us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:18.976159 (+ 1560us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:18.976786 (+ 627us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:18.980782 (+ 3996us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:18.980859 (+ 77us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:18.980881 (+ 22us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:18.980922 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:18.984046 (+ 3124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:18.984844 (+ 798us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:18.984851 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:18.984858 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:18.984957 (+ 99us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:18.985338 (+ 381us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:18.985342 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:18.985344 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:18.985813 (+ 469us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:18.986134 (+ 321us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:18.986588 (+ 454us) server_negotiation.cc:300] Negotiation successful
1027 14:57:18.986883 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":606,"thread_start_us":395,"threads_started":1}
I20251027 14:57:18.989548 22675 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3814c3b95c3842aca80003a5a7457abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20251027 14:57:18.989936 22675 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:18.990047 22675 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3814c3b95c3842aca80003a5a7457abe, State: Initialized, Role: FOLLOWER
I20251027 14:57:18.990635 22675 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3814c3b95c3842aca80003a5a7457abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20251027 14:57:18.990825 22675 raft_consensus.cc:399] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:18.990950 22675 raft_consensus.cc:493] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:18.991108 22675 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:18.991816 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:18.993724 22675 raft_consensus.cc:515] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3814c3b95c3842aca80003a5a7457abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20251027 14:57:18.994207 22675 leader_election.cc:304] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 3814c3b95c3842aca80003a5a7457abe; no voters:
I20251027 14:57:18.994824 22675 leader_election.cc:290] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:18.994932 22680 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:18.996595 22680 raft_consensus.cc:697] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [term 1 LEADER]: Becoming Leader. State: Replica: 3814c3b95c3842aca80003a5a7457abe, State: Running, Role: LEADER
I20251027 14:57:18.997152 22675 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:18.997098 22680 consensus_queue.cc:237] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3814c3b95c3842aca80003a5a7457abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } }
I20251027 14:57:19.003185 22684 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "3814c3b95c3842aca80003a5a7457abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3814c3b95c3842aca80003a5a7457abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } } }
I20251027 14:57:19.003868 22684 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:19.007342 22685 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 3814c3b95c3842aca80003a5a7457abe. Latest consensus state: current_term: 1 leader_uuid: "3814c3b95c3842aca80003a5a7457abe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "3814c3b95c3842aca80003a5a7457abe" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 36607 } } }
I20251027 14:57:19.007632 22693 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:19.007952 22685 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:19.018023 22693 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:19.019140 22678 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:18.997934 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:39963 (local address 127.0.0.1:36607)
1027 14:57:18.998105 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:18.998112 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:19.001176 (+ 3064us) server_negotiation.cc:408] Connection header received
1027 14:57:19.001374 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:19.001382 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:19.001468 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:19.001564 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:19.004229 (+ 2665us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:19.005690 (+ 1461us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:19.007359 (+ 1669us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:19.007960 (+ 601us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:19.011271 (+ 3311us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:19.011311 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:19.011318 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:19.011380 (+ 62us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:19.015342 (+ 3962us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:19.016363 (+ 1021us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:19.016369 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:19.016399 (+ 30us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:19.016492 (+ 93us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:19.016970 (+ 478us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:19.016975 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:19.016978 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:19.017247 (+ 269us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:19.017563 (+ 316us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:19.018579 (+ 1016us) server_negotiation.cc:300] Negotiation successful
1027 14:57:19.018786 (+ 207us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:57:19.026865 22693 catalog_manager.cc:1357] Generated new cluster ID: de8d4dab7c3642d5afdfc42d3dbc291a
I20251027 14:57:19.026968 22693 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:19.041126 22693 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:19.042268 22693 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:19.054574 22693 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 3814c3b95c3842aca80003a5a7457abe: Generated new TSK 0
I20251027 14:57:19.055351 22693 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:19.060671 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22608
I20251027 14:57:19.062151 22670 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:19.150403 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22608
W20251027 14:57:19.164767 22590 connection.cc:537] client connection to 127.0.0.1:36607 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:36607): Connection reset by peer (error 104)
2025-10-27T14:57:19Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/9 (2824 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/10
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:19 dist-test-slave-hxxt krb5kdc[22707](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:19 dist-test-slave-hxxt krb5kdc[22707](info): set up 2 sockets
Oct 27 14:57:19 dist-test-slave-hxxt krb5kdc[22707](info): commencing operation
krb5kdc: starting...
W20251027 14:57:21.040567 416 mini_kdc.cc:121] Time spent starting KDC: real 1.857s user 0.003s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:21 dist-test-slave-hxxt krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577041, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:21Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:21.332576 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38781
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:33245
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:38781
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:21.659909 22723 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:21.660220 22723 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:21.660351 22723 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:21.668358 22723 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:21.668505 22723 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:21.668596 22723 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:21.668649 22723 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:21.668726 22723 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:21.679308 22723 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:33245
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:38781
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:38781
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:21.680886 22723 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:21.682186 22723 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:21.690176 22728 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:21.690176 22729 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:21.690562 22731 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:21.691108 22723 server_base.cc:1047] running on GCE node
I20251027 14:57:21.691857 22723 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:21.693286 22723 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:21.694494 22723 hybrid_clock.cc:648] HybridClock initialized: now 1761577041694425 us; error 44 us; skew 500 ppm
I20251027 14:57:21.694813 22723 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:21 dist-test-slave-hxxt krb5kdc[22707](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577041, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:21.697773 22723 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:21.699257 22723 webserver.cc:492] Webserver started at http://127.0.0.1:38811/ using document root <none> and password file <none>
I20251027 14:57:21.699970 22723 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:21.700057 22723 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:21.700470 22723 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:21.702981 22723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "7628d76139144d1fba117c586e7a0dd9"
format_stamp: "Formatted at 2025-10-27 14:57:21 on dist-test-slave-hxxt"
server_key: "4c94b5377f3bd7f3361a7a83b39af1f8"
server_key_iv: "a8a24418d5f77fcb3d79e3d7e544ede9"
server_key_version: "encryptionkey@0"
I20251027 14:57:21.703756 22723 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "7628d76139144d1fba117c586e7a0dd9"
format_stamp: "Formatted at 2025-10-27 14:57:21 on dist-test-slave-hxxt"
server_key: "4c94b5377f3bd7f3361a7a83b39af1f8"
server_key_iv: "a8a24418d5f77fcb3d79e3d7e544ede9"
server_key_version: "encryptionkey@0"
I20251027 14:57:21.708952 22723 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20251027 14:57:21.712209 22738 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:21.713583 22723 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:57:21.713743 22723 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "7628d76139144d1fba117c586e7a0dd9"
format_stamp: "Formatted at 2025-10-27 14:57:21 on dist-test-slave-hxxt"
server_key: "4c94b5377f3bd7f3361a7a83b39af1f8"
server_key_iv: "a8a24418d5f77fcb3d79e3d7e544ede9"
server_key_version: "encryptionkey@0"
I20251027 14:57:21.713948 22723 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:21.742010 22723 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:21.748831 22723 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:21.749116 22723 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:21.768085 22723 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:38781
I20251027 14:57:21.768103 22789 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:38781 every 8 connection(s)
I20251027 14:57:21.769591 22723 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:21.773787 22790 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:21.775578 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22723
I20251027 14:57:21.775786 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_10.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:21.776259 416 external_mini_cluster.cc:1468] Setting key 66be9f1d5511fdd91c3050a999b0dbd2
I20251027 14:57:21.786113 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9: Bootstrap starting.
I20251027 14:57:21.788668 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:21.778097 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:57807 (local address 127.0.0.1:38781)
1027 14:57:21.778898 (+ 801us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:21.778913 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:21.778963 (+ 50us) server_negotiation.cc:408] Connection header received
1027 14:57:21.780143 (+ 1180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:21.780166 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:21.780630 (+ 464us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:21.780976 (+ 346us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:21.781865 (+ 889us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:21.783254 (+ 1389us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:21.785011 (+ 1757us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:21.785469 (+ 458us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:21.786155 (+ 686us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:21.786273 (+ 118us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:21.786296 (+ 23us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:21.786865 (+ 569us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:21.786917 (+ 52us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:21.786947 (+ 30us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:21.787092 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:21.787410 (+ 318us) server_negotiation.cc:300] Negotiation successful
1027 14:57:21.787615 (+ 205us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":610,"thread_start_us":411,"threads_started":1}
I20251027 14:57:21.789620 22790 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:21.790941 22790 log.cc:826] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:21.792909 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:21.793964 22790 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9: No bootstrap required, opened a new log
I20251027 14:57:21.800001 22790 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7628d76139144d1fba117c586e7a0dd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20251027 14:57:21.800501 22790 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:21.800618 22790 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7628d76139144d1fba117c586e7a0dd9, State: Initialized, Role: FOLLOWER
I20251027 14:57:21.801378 22790 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7628d76139144d1fba117c586e7a0dd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20251027 14:57:21.801651 22790 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:21.801819 22790 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:21.802052 22790 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:21.803364 22793 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:21.797296 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:45117 (local address 127.0.0.1:38781)
1027 14:57:21.797495 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:21.797502 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:21.797831 (+ 329us) server_negotiation.cc:408] Connection header received
1027 14:57:21.797998 (+ 167us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:21.798002 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:21.798065 (+ 63us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:21.798158 (+ 93us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:21.799065 (+ 907us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:21.799933 (+ 868us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:21.801582 (+ 1649us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:21.802155 (+ 573us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:21.802587 (+ 432us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:21.802655 (+ 68us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:21.802661 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:21.802681 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:21.802730 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:21.802752 (+ 22us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:21.802837 (+ 85us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:21.803001 (+ 164us) server_negotiation.cc:300] Negotiation successful
1027 14:57:21.803051 (+ 50us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
I20251027 14:57:21.804888 22790 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7628d76139144d1fba117c586e7a0dd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20251027 14:57:21.805405 22790 leader_election.cc:304] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7628d76139144d1fba117c586e7a0dd9; no voters:
I20251027 14:57:21.805994 22790 leader_election.cc:290] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:21.806116 22800 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:21.806794 22800 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [term 1 LEADER]: Becoming Leader. State: Replica: 7628d76139144d1fba117c586e7a0dd9, State: Running, Role: LEADER
I20251027 14:57:21.807271 22800 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7628d76139144d1fba117c586e7a0dd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } }
I20251027 14:57:21.808017 22790 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:21.810246 22802 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7628d76139144d1fba117c586e7a0dd9. Latest consensus state: current_term: 1 leader_uuid: "7628d76139144d1fba117c586e7a0dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7628d76139144d1fba117c586e7a0dd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } } }
I20251027 14:57:21.810850 22802 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:21.810539 22801 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7628d76139144d1fba117c586e7a0dd9" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7628d76139144d1fba117c586e7a0dd9" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 38781 } } }
I20251027 14:57:21.811686 22801 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:21.811858 22808 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:21.818279 22808 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:21.829034 22808 catalog_manager.cc:1357] Generated new cluster ID: bd7a86b34cba418a914577850d5ce098
I20251027 14:57:21.829154 22808 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:21.851351 22808 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:21.852674 22808 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:21.862417 22808 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 7628d76139144d1fba117c586e7a0dd9: Generated new TSK 0
I20251027 14:57:21.863273 22808 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:21.877434 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22723
W20251027 14:57:21.878643 22785 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.25:57807
W20251027 14:57:21.879455 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22723: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:21.879576 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22723
W20251027 14:57:21.893561 22705 connection.cc:537] client connection to 127.0.0.1:38781 recv error: Network error: failed to read from TLS socket (remote: 127.0.0.1:38781): Connection reset by peer (error 104)
2025-10-27T14:57:21Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/10 (2728 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/11
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:21 dist-test-slave-hxxt krb5kdc[22821](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:21 dist-test-slave-hxxt krb5kdc[22821](info): set up 2 sockets
Oct 27 14:57:21 dist-test-slave-hxxt krb5kdc[22821](info): commencing operation
krb5kdc: starting...
W20251027 14:57:23.776599 416 mini_kdc.cc:121] Time spent starting KDC: real 1.864s user 0.001s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:23 dist-test-slave-hxxt krb5kdc[22821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577043, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:24.064010 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33053
--webserver_interface=127.0.0.1
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.0.1:38073
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.0.1:33053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:24.391412 22837 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:24.391718 22837 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:24.391819 22837 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:24.400914 22837 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:24.401041 22837 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:24.401131 22837 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:24.401203 22837 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:24.401296 22837 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:24.412513 22837 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.0.1:38073
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.0.1:33053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.0.1
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.0.1:33053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.0.1
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:24.414059 22837 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:24.415680 22837 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:24.424281 22845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:24.424432 22842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:24.424335 22843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:24.425040 22837 server_base.cc:1047] running on GCE node
I20251027 14:57:24.425784 22837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:24.427182 22837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:24.428448 22837 hybrid_clock.cc:648] HybridClock initialized: now 1761577044428346 us; error 48 us; skew 500 ppm
I20251027 14:57:24.428788 22837 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:24 dist-test-slave-hxxt krb5kdc[22821](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577044, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:24.431958 22837 init.cc:377] Logged in from keytab as kudu/127.0.0.1@KRBTEST.COM (short username kudu)
I20251027 14:57:24.433565 22837 webserver.cc:492] Webserver started at http://127.0.0.1:40997/ using document root <none> and password file <none>
I20251027 14:57:24.434319 22837 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:24.434404 22837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:24.434770 22837 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:24.437466 22837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "39c4c3502235447bb045535e33b9e362"
format_stamp: "Formatted at 2025-10-27 14:57:24 on dist-test-slave-hxxt"
server_key: "5c3635de86bbaf49389f9816b10e6014"
server_key_iv: "3be04b2f7dba5ca49e4637c4aaf8f3e6"
server_key_version: "encryptionkey@0"
I20251027 14:57:24.438242 22837 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "39c4c3502235447bb045535e33b9e362"
format_stamp: "Formatted at 2025-10-27 14:57:24 on dist-test-slave-hxxt"
server_key: "5c3635de86bbaf49389f9816b10e6014"
server_key_iv: "3be04b2f7dba5ca49e4637c4aaf8f3e6"
server_key_version: "encryptionkey@0"
I20251027 14:57:24.443612 22837 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20251027 14:57:24.446919 22852 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:24.448205 22837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:57:24.448369 22837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "39c4c3502235447bb045535e33b9e362"
format_stamp: "Formatted at 2025-10-27 14:57:24 on dist-test-slave-hxxt"
server_key: "5c3635de86bbaf49389f9816b10e6014"
server_key_iv: "3be04b2f7dba5ca49e4637c4aaf8f3e6"
server_key_version: "encryptionkey@0"
I20251027 14:57:24.448632 22837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:24.454624 22837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:24.458667 22837 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:24.458916 22837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:24.477514 22837 rpc_server.cc:307] RPC server started. Bound to: 127.0.0.1:33053
I20251027 14:57:24.477533 22903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.0.1:33053 every 8 connection(s)
I20251027 14:57:24.478965 22837 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:24.483026 22904 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:24.487329 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22837
I20251027 14:57:24.487534 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_11.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:24.488005 416 external_mini_cluster.cc:1468] Setting key 761c1ff4ac91856312b5b23c9b244a3e
I20251027 14:57:24.494925 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362: Bootstrap starting.
I20251027 14:57:24.495734 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:24.489751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:52265 (local address 127.0.0.1:33053)
1027 14:57:24.490503 (+ 752us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:24.490516 (+ 13us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:24.490562 (+ 46us) server_negotiation.cc:408] Connection header received
1027 14:57:24.492250 (+ 1688us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:24.492280 (+ 30us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:24.492724 (+ 444us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:24.493108 (+ 384us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:24.494199 (+ 1091us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:24.494206 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:24.494220 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:24.494233 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:24.494282 (+ 49us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:24.494300 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:24.494468 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:24.494624 (+ 156us) server_negotiation.cc:300] Negotiation successful
1027 14:57:24.494778 (+ 154us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":566,"thread_start_us":376,"threads_started":1}
I20251027 14:57:24.498149 22904 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:24.499259 22904 log.cc:826] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:24.499487 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:24.502509 22904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362: No bootstrap required, opened a new log
I20251027 14:57:24.509527 22904 raft_consensus.cc:359] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c4c3502235447bb045535e33b9e362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20251027 14:57:24.509924 22904 raft_consensus.cc:385] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:24.510010 22904 raft_consensus.cc:740] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39c4c3502235447bb045535e33b9e362, State: Initialized, Role: FOLLOWER
I20251027 14:57:24.510761 22904 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c4c3502235447bb045535e33b9e362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20251027 14:57:24.510983 22904 raft_consensus.cc:399] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:24.511096 22904 raft_consensus.cc:493] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:24.511238 22904 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:24.514612 22904 raft_consensus.cc:515] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c4c3502235447bb045535e33b9e362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20251027 14:57:24.515242 22904 leader_election.cc:304] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 39c4c3502235447bb045535e33b9e362; no voters:
I20251027 14:57:24.515995 22904 leader_election.cc:290] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:24.516122 22914 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:24.516405 22907 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:24.513475 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 10.240.0.25:46043 (local address 127.0.0.1:33053)
1027 14:57:24.513701 (+ 226us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:24.513708 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:24.514238 (+ 530us) server_negotiation.cc:408] Connection header received
1027 14:57:24.514434 (+ 196us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:24.514438 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:24.514489 (+ 51us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:24.514571 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:24.515604 (+ 1033us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:24.515609 (+ 5us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:24.515612 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:24.515628 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:24.515675 (+ 47us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:24.515691 (+ 16us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:24.515819 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:24.515956 (+ 137us) server_negotiation.cc:300] Negotiation successful
1027 14:57:24.516028 (+ 72us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":89}
I20251027 14:57:24.518940 22904 sys_catalog.cc:565] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:24.520704 22914 raft_consensus.cc:697] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [term 1 LEADER]: Becoming Leader. State: Replica: 39c4c3502235447bb045535e33b9e362, State: Running, Role: LEADER
I20251027 14:57:24.521350 22914 consensus_queue.cc:237] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c4c3502235447bb045535e33b9e362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } }
I20251027 14:57:24.524735 22915 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "39c4c3502235447bb045535e33b9e362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c4c3502235447bb045535e33b9e362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } } }
I20251027 14:57:24.525100 22915 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:24.525750 22916 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 39c4c3502235447bb045535e33b9e362. Latest consensus state: current_term: 1 leader_uuid: "39c4c3502235447bb045535e33b9e362" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "39c4c3502235447bb045535e33b9e362" member_type: VOTER last_known_addr { host: "127.0.0.1" port: 33053 } } }
I20251027 14:57:24.526011 22916 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:24.526153 22922 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:24.531700 22922 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:24.540529 22922 catalog_manager.cc:1357] Generated new cluster ID: 392f50f9ba524ea3bc040e638e3e9b2d
I20251027 14:57:24.540661 22922 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:24.569224 22922 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:24.569607 22922 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:24.588644 22922 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 39c4c3502235447bb045535e33b9e362: Generated new TSK 0
I20251027 14:57:24.589527 22922 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:24.603667 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22837
W20251027 14:57:24.604759 22899 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 10.240.0.25:52265
W20251027 14:57:24.605468 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22837: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:24.605590 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22837
2025-10-27T14:57:24Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/11 (2726 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/12
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:24 dist-test-slave-hxxt krb5kdc[22935](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:24 dist-test-slave-hxxt krb5kdc[22935](info): set up 2 sockets
Oct 27 14:57:24 dist-test-slave-hxxt krb5kdc[22935](info): commencing operation
krb5kdc: starting...
W20251027 14:57:26.492322 416 mini_kdc.cc:121] Time spent starting KDC: real 1.853s user 0.005s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:26 dist-test-slave-hxxt krb5kdc[22935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:26Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:26Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:26.781589 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34053
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:42725
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34053
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:27.111238 22951 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:27.111552 22951 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:27.111687 22951 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:27.119627 22951 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:27.119747 22951 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:27.119837 22951 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:27.119899 22951 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:27.119983 22951 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:27.130345 22951 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:42725
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34053
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34053
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:27.131934 22951 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:27.133227 22951 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:27.141232 22956 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:27.141178 22957 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:27.141727 22959 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:27.142506 22951 server_base.cc:1047] running on GCE node
I20251027 14:57:27.143239 22951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:27.144651 22951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:27.145881 22951 hybrid_clock.cc:648] HybridClock initialized: now 1761577047145820 us; error 39 us; skew 500 ppm
I20251027 14:57:27.146204 22951 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:27 dist-test-slave-hxxt krb5kdc[22935](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577047, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:27.149461 22951 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:27.150909 22951 webserver.cc:492] Webserver started at http://127.0.104.62:37343/ using document root <none> and password file <none>
I20251027 14:57:27.151600 22951 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:27.151719 22951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:27.152132 22951 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:27.154858 22951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "d8f8c7e0868f475b8bc708e5ac582259"
format_stamp: "Formatted at 2025-10-27 14:57:27 on dist-test-slave-hxxt"
server_key: "f98211d44a496c2614355a985133cf77"
server_key_iv: "bb7f36b0e765f374f32b49e21e635785"
server_key_version: "encryptionkey@0"
I20251027 14:57:27.155640 22951 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "d8f8c7e0868f475b8bc708e5ac582259"
format_stamp: "Formatted at 2025-10-27 14:57:27 on dist-test-slave-hxxt"
server_key: "f98211d44a496c2614355a985133cf77"
server_key_iv: "bb7f36b0e765f374f32b49e21e635785"
server_key_version: "encryptionkey@0"
I20251027 14:57:27.161000 22951 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20251027 14:57:27.164309 22966 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:27.165647 22951 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:57:27.165807 22951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "d8f8c7e0868f475b8bc708e5ac582259"
format_stamp: "Formatted at 2025-10-27 14:57:27 on dist-test-slave-hxxt"
server_key: "f98211d44a496c2614355a985133cf77"
server_key_iv: "bb7f36b0e765f374f32b49e21e635785"
server_key_version: "encryptionkey@0"
I20251027 14:57:27.166004 22951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:27.183061 22951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:27.190819 22951 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:27.191069 22951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:27.209695 22951 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34053
I20251027 14:57:27.209702 23017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34053 every 8 connection(s)
I20251027 14:57:27.211241 22951 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:27.214701 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 22951
I20251027 14:57:27.214895 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_12.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:27.215288 416 external_mini_cluster.cc:1468] Setting key d3a83bfe6063460c3e1f70b27b19e55d
I20251027 14:57:27.215595 23018 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Oct 27 14:57:27 dist-test-slave-hxxt krb5kdc[22935](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577046, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:27.229259 23018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259: Bootstrap starting.
I20251027 14:57:27.232667 23018 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:27.233726 23018 log.cc:826] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:27.236810 23018 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259: No bootstrap required, opened a new log
I20251027 14:57:27.236840 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:27.217030 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:40925 (local address 127.0.104.62:34053)
1027 14:57:27.217786 (+ 756us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:27.217802 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:27.217869 (+ 67us) server_negotiation.cc:408] Connection header received
1027 14:57:27.219337 (+ 1468us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:27.219369 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:27.219975 (+ 606us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:27.220525 (+ 550us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:27.222129 (+ 1604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:27.224153 (+ 2024us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:27.225383 (+ 1230us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:27.226006 (+ 623us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:27.229486 (+ 3480us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:27.229583 (+ 97us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:27.229601 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:27.229645 (+ 44us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:27.232686 (+ 3041us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:27.233495 (+ 809us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:27.233505 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:27.233513 (+ 8us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:27.233629 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:27.234092 (+ 463us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:27.234096 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:27.234098 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:27.234616 (+ 518us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:27.234993 (+ 377us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:27.235457 (+ 464us) server_negotiation.cc:300] Negotiation successful
1027 14:57:27.235771 (+ 314us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":542,"thread_start_us":320,"threads_started":1}
I20251027 14:57:27.241111 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:27.242810 23018 raft_consensus.cc:359] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f8c7e0868f475b8bc708e5ac582259" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34053 } }
I20251027 14:57:27.243170 23018 raft_consensus.cc:385] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:27.243266 23018 raft_consensus.cc:740] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d8f8c7e0868f475b8bc708e5ac582259, State: Initialized, Role: FOLLOWER
I20251027 14:57:27.243894 23018 consensus_queue.cc:260] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f8c7e0868f475b8bc708e5ac582259" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34053 } }
I20251027 14:57:27.244145 23018 raft_consensus.cc:399] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:27.244280 23018 raft_consensus.cc:493] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:27.244447 23018 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:27.247040 23018 raft_consensus.cc:515] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f8c7e0868f475b8bc708e5ac582259" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34053 } }
I20251027 14:57:27.247556 23018 leader_election.cc:304] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d8f8c7e0868f475b8bc708e5ac582259; no voters:
I20251027 14:57:27.248289 23018 leader_election.cc:290] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:27.248423 23028 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:27.249750 23028 raft_consensus.cc:697] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [term 1 LEADER]: Becoming Leader. State: Replica: d8f8c7e0868f475b8bc708e5ac582259, State: Running, Role: LEADER
I20251027 14:57:27.250303 23028 consensus_queue.cc:237] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f8c7e0868f475b8bc708e5ac582259" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34053 } }
I20251027 14:57:27.250710 23018 sys_catalog.cc:565] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:27.254086 23030 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d8f8c7e0868f475b8bc708e5ac582259. Latest consensus state: current_term: 1 leader_uuid: "d8f8c7e0868f475b8bc708e5ac582259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f8c7e0868f475b8bc708e5ac582259" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34053 } } }
I20251027 14:57:27.254124 23029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d8f8c7e0868f475b8bc708e5ac582259" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d8f8c7e0868f475b8bc708e5ac582259" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34053 } } }
I20251027 14:57:27.254453 23030 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:27.254453 23029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:27.255793 23035 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:27.263362 23021 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:27.245332 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:45003 (local address 127.0.104.62:34053)
1027 14:57:27.245476 (+ 144us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:27.245481 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:27.246017 (+ 536us) server_negotiation.cc:408] Connection header received
1027 14:57:27.246175 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:27.246181 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:27.246246 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:27.246328 (+ 82us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:27.247975 (+ 1647us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:27.249414 (+ 1439us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:27.250950 (+ 1536us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:27.251847 (+ 897us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:27.253969 (+ 2122us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:27.254009 (+ 40us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:27.254014 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:27.254059 (+ 45us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:27.258670 (+ 4611us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:27.259934 (+ 1264us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:27.259941 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:27.259945 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:27.260061 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:27.260633 (+ 572us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:27.260638 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:27.260643 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:27.261000 (+ 357us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:27.262516 (+ 1516us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:27.262711 (+ 195us) server_negotiation.cc:300] Negotiation successful
1027 14:57:27.262933 (+ 222us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:57:27.270455 23035 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:27.279449 23035 catalog_manager.cc:1357] Generated new cluster ID: a231b8c8eeb5420c8fe697885c4f6903
I20251027 14:57:27.279573 23035 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:27.291678 23035 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:27.293444 23035 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:27.323246 23035 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d8f8c7e0868f475b8bc708e5ac582259: Generated new TSK 0
I20251027 14:57:27.324362 23035 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:27.338868 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 22951
I20251027 14:57:27.340529 23013 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:27.433279 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 22951
W20251027 14:57:27.447444 22933 connection.cc:537] client connection to 127.0.104.62:34053 recv error: Network error: failed to read from TLS socket (remote: 127.0.104.62:34053): Connection reset by peer (error 104)
2025-10-27T14:57:27Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/12 (2826 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/13
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:27 dist-test-slave-hxxt krb5kdc[23050](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:27 dist-test-slave-hxxt krb5kdc[23050](info): set up 2 sockets
Oct 27 14:57:27 dist-test-slave-hxxt krb5kdc[23050](info): commencing operation
krb5kdc: starting...
W20251027 14:57:29.336470 416 mini_kdc.cc:121] Time spent starting KDC: real 1.870s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:29 dist-test-slave-hxxt krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577049, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:29Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:29.629396 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44915
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:40787
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:44915
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:29.955502 23066 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:29.955864 23066 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:29.956010 23066 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:29.964237 23066 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:29.964357 23066 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:29.964459 23066 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:29.964520 23066 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:29.964670 23066 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:29.975934 23066 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:40787
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:44915
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:44915
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:29.977596 23066 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:29.978965 23066 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:29.986939 23072 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:29.987385 23071 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:29.987555 23074 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:29.988076 23066 server_base.cc:1047] running on GCE node
I20251027 14:57:29.988914 23066 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:29.990291 23066 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:29.991511 23066 hybrid_clock.cc:648] HybridClock initialized: now 1761577049991444 us; error 41 us; skew 500 ppm
I20251027 14:57:29.991853 23066 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:29 dist-test-slave-hxxt krb5kdc[23050](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577049, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:29.994922 23066 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:29.996541 23066 webserver.cc:492] Webserver started at http://127.0.104.62:36689/ using document root <none> and password file <none>
I20251027 14:57:29.997277 23066 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:29.997370 23066 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:29.997732 23066 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:30.000257 23066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "1a801b22e55849bf9bd1101276e1fdda"
format_stamp: "Formatted at 2025-10-27 14:57:29 on dist-test-slave-hxxt"
server_key: "997548c0625ee7397ac285dcc364714f"
server_key_iv: "ec04273c6d8dbba7e427df9e6d3a0250"
server_key_version: "encryptionkey@0"
I20251027 14:57:30.001092 23066 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "1a801b22e55849bf9bd1101276e1fdda"
format_stamp: "Formatted at 2025-10-27 14:57:29 on dist-test-slave-hxxt"
server_key: "997548c0625ee7397ac285dcc364714f"
server_key_iv: "ec04273c6d8dbba7e427df9e6d3a0250"
server_key_version: "encryptionkey@0"
I20251027 14:57:30.006474 23066 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:57:30.009801 23081 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:30.011193 23066 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:57:30.011351 23066 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "1a801b22e55849bf9bd1101276e1fdda"
format_stamp: "Formatted at 2025-10-27 14:57:29 on dist-test-slave-hxxt"
server_key: "997548c0625ee7397ac285dcc364714f"
server_key_iv: "ec04273c6d8dbba7e427df9e6d3a0250"
server_key_version: "encryptionkey@0"
I20251027 14:57:30.011550 23066 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:30.027854 23066 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:30.031647 23066 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:30.031898 23066 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:30.050563 23132 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:44915 every 8 connection(s)
I20251027 14:57:30.050570 23066 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:44915
I20251027 14:57:30.052035 23066 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:30.053310 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23066
I20251027 14:57:30.053503 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_13.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:30.053999 416 external_mini_cluster.cc:1468] Setting key b35f62ea4874cd1350e8aff6e94e5b65
I20251027 14:57:30.057010 23133 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:30.066087 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:30.055758 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:44721 (local address 127.0.104.62:44915)
1027 14:57:30.056530 (+ 772us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:30.056542 (+ 12us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:30.056594 (+ 52us) server_negotiation.cc:408] Connection header received
1027 14:57:30.057968 (+ 1374us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:30.057993 (+ 25us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:30.058394 (+ 401us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:30.058759 (+ 365us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:30.059641 (+ 882us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:30.061075 (+ 1434us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:30.062678 (+ 1603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:30.063104 (+ 426us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:30.063723 (+ 619us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:30.063809 (+ 86us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:30.063824 (+ 15us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:30.064367 (+ 543us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:30.064457 (+ 90us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:30.064481 (+ 24us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:30.064617 (+ 136us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:30.064866 (+ 249us) server_negotiation.cc:300] Negotiation successful
1027 14:57:30.065079 (+ 213us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":567,"thread_start_us":354,"threads_started":1}
I20251027 14:57:30.068673 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda: Bootstrap starting.
I20251027 14:57:30.070039 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:30.072217 23133 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:30.073597 23133 log.cc:826] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:30.077172 23133 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda: No bootstrap required, opened a new log
I20251027 14:57:30.081359 23136 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:30.075182 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:37717 (local address 127.0.104.62:44915)
1027 14:57:30.075369 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:30.075376 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:30.075714 (+ 338us) server_negotiation.cc:408] Connection header received
1027 14:57:30.075912 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:30.075917 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:30.075975 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:30.076054 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:30.076975 (+ 921us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:30.077827 (+ 852us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:30.079486 (+ 1659us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:30.079935 (+ 449us) server_negotiation.cc:663] Negotiated TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:30.080482 (+ 547us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:30.080511 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:30.080515 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:30.080532 (+ 17us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:30.080574 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:30.080588 (+ 14us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:30.080658 (+ 70us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:30.080873 (+ 215us) server_negotiation.cc:300] Negotiation successful
1027 14:57:30.080926 (+ 53us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
I20251027 14:57:30.084764 23133 raft_consensus.cc:359] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a801b22e55849bf9bd1101276e1fdda" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44915 } }
I20251027 14:57:30.085119 23133 raft_consensus.cc:385] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:30.085223 23133 raft_consensus.cc:740] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 1a801b22e55849bf9bd1101276e1fdda, State: Initialized, Role: FOLLOWER
I20251027 14:57:30.085801 23133 consensus_queue.cc:260] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a801b22e55849bf9bd1101276e1fdda" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44915 } }
I20251027 14:57:30.085991 23133 raft_consensus.cc:399] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:30.086109 23133 raft_consensus.cc:493] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:30.086256 23133 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:30.088670 23133 raft_consensus.cc:515] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a801b22e55849bf9bd1101276e1fdda" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44915 } }
I20251027 14:57:30.089159 23133 leader_election.cc:304] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 1a801b22e55849bf9bd1101276e1fdda; no voters:
I20251027 14:57:30.089710 23133 leader_election.cc:290] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:30.089833 23143 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:30.090471 23143 raft_consensus.cc:697] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [term 1 LEADER]: Becoming Leader. State: Replica: 1a801b22e55849bf9bd1101276e1fdda, State: Running, Role: LEADER
I20251027 14:57:30.090997 23143 consensus_queue.cc:237] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a801b22e55849bf9bd1101276e1fdda" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44915 } }
I20251027 14:57:30.091665 23133 sys_catalog.cc:565] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:30.093703 23145 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [sys.catalog]: SysCatalogTable state changed. Reason: New leader 1a801b22e55849bf9bd1101276e1fdda. Latest consensus state: current_term: 1 leader_uuid: "1a801b22e55849bf9bd1101276e1fdda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a801b22e55849bf9bd1101276e1fdda" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44915 } } }
I20251027 14:57:30.094013 23145 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:30.094234 23144 sys_catalog.cc:455] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "1a801b22e55849bf9bd1101276e1fdda" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "1a801b22e55849bf9bd1101276e1fdda" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 44915 } } }
I20251027 14:57:30.094542 23144 sys_catalog.cc:458] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:30.095244 23149 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:30.099964 23149 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:30.109387 23149 catalog_manager.cc:1357] Generated new cluster ID: 380c070dacf84ba5840c69d3e7e94817
I20251027 14:57:30.109508 23149 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:30.138736 23149 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:30.139981 23149 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:30.147559 23149 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 1a801b22e55849bf9bd1101276e1fdda: Generated new TSK 0
I20251027 14:57:30.148623 23149 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:30.161571 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23066
W20251027 14:57:30.162851 23128 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:44721
W20251027 14:57:30.163591 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23066: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:30.163738 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23066
W20251027 14:57:30.177641 23048 connection.cc:537] client connection to 127.0.104.62:44915 recv error: Network error: failed to read from TLS socket (remote: 127.0.104.62:44915): Connection reset by peer (error 104)
2025-10-27T14:57:30Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/13 (2730 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/14
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:30 dist-test-slave-hxxt krb5kdc[23164](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:30 dist-test-slave-hxxt krb5kdc[23164](info): set up 2 sockets
Oct 27 14:57:30 dist-test-slave-hxxt krb5kdc[23164](info): commencing operation
krb5kdc: starting...
W20251027 14:57:32.052611 416 mini_kdc.cc:121] Time spent starting KDC: real 1.856s user 0.005s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:32 dist-test-slave-hxxt krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577052, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:32Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:32Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:32.340441 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37079
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:35751
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:37079
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=true with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:32.665640 23180 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:32.665935 23180 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:32.666061 23180 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:32.674105 23180 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:32.674212 23180 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:32.674301 23180 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:32.674362 23180 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:32.674450 23180 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:32.684973 23180 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:35751
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:37079
--rpc_encrypt_loopback_connections=true
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:37079
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:32.686514 23180 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:32.687831 23180 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:32.695631 23185 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:32.695559 23186 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:32.696406 23188 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:32.696635 23180 server_base.cc:1047] running on GCE node
I20251027 14:57:32.697373 23180 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:32.698726 23180 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:32.699932 23180 hybrid_clock.cc:648] HybridClock initialized: now 1761577052699871 us; error 43 us; skew 500 ppm
I20251027 14:57:32.700237 23180 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:32 dist-test-slave-hxxt krb5kdc[23164](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577052, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:32.703267 23180 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:32.704846 23180 webserver.cc:492] Webserver started at http://127.0.104.62:44279/ using document root <none> and password file <none>
I20251027 14:57:32.705557 23180 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:32.705644 23180 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:32.705992 23180 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:32.708618 23180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "ab5e4bd8af46498fbdf7dbdf91607ada"
format_stamp: "Formatted at 2025-10-27 14:57:32 on dist-test-slave-hxxt"
server_key: "b009e5e9a95d6fa09536e115c8935d2e"
server_key_iv: "e4e8251361ccf80b3c83bc9c6688937a"
server_key_version: "encryptionkey@0"
I20251027 14:57:32.709384 23180 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "ab5e4bd8af46498fbdf7dbdf91607ada"
format_stamp: "Formatted at 2025-10-27 14:57:32 on dist-test-slave-hxxt"
server_key: "b009e5e9a95d6fa09536e115c8935d2e"
server_key_iv: "e4e8251361ccf80b3c83bc9c6688937a"
server_key_version: "encryptionkey@0"
I20251027 14:57:32.714681 23180 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:57:32.717969 23195 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:32.719218 23180 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:57:32.719369 23180 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "ab5e4bd8af46498fbdf7dbdf91607ada"
format_stamp: "Formatted at 2025-10-27 14:57:32 on dist-test-slave-hxxt"
server_key: "b009e5e9a95d6fa09536e115c8935d2e"
server_key_iv: "e4e8251361ccf80b3c83bc9c6688937a"
server_key_version: "encryptionkey@0"
I20251027 14:57:32.719564 23180 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:32.726060 23180 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:32.734256 23180 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:32.734526 23180 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:32.753428 23180 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:37079
I20251027 14:57:32.753448 23246 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:37079 every 8 connection(s)
I20251027 14:57:32.754825 23180 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:32.758880 23247 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:32.762212 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23180
I20251027 14:57:32.762413 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_14.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:32.762918 416 external_mini_cluster.cc:1468] Setting key 9a23cfc38377458abf1ccb3fe2b97704
I20251027 14:57:32.769897 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:32.764726 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:49333 (local address 127.0.104.62:37079)
1027 14:57:32.765424 (+ 698us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:32.765435 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:32.765487 (+ 52us) server_negotiation.cc:408] Connection header received
1027 14:57:32.766607 (+ 1120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:32.766629 (+ 22us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:32.766967 (+ 338us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:32.767382 (+ 415us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:32.768452 (+ 1070us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:32.768459 (+ 7us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:32.768477 (+ 18us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:32.768490 (+ 13us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:32.768532 (+ 42us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:32.768553 (+ 21us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:32.768751 (+ 198us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:32.768893 (+ 142us) server_negotiation.cc:300] Negotiation successful
1027 14:57:32.769056 (+ 163us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":523,"thread_start_us":331,"threads_started":1}
I20251027 14:57:32.770196 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada: Bootstrap starting.
I20251027 14:57:32.773391 23247 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:32.773557 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:32.774382 23247 log.cc:826] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:32.777135 23247 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada: No bootstrap required, opened a new log
I20251027 14:57:32.780108 23250 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:32.777879 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35339 (local address 127.0.104.62:37079)
1027 14:57:32.778042 (+ 163us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:32.778048 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:32.778376 (+ 328us) server_negotiation.cc:408] Connection header received
1027 14:57:32.778508 (+ 132us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:32.778514 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:32.778572 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:32.778689 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:32.779545 (+ 856us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:32.779551 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:32.779556 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:32.779576 (+ 20us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:32.779621 (+ 45us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:32.779639 (+ 18us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:32.779755 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:32.779795 (+ 40us) server_negotiation.cc:300] Negotiation successful
1027 14:57:32.779844 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":58}
I20251027 14:57:32.782871 23247 raft_consensus.cc:359] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37079 } }
I20251027 14:57:32.783187 23247 raft_consensus.cc:385] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:32.783285 23247 raft_consensus.cc:740] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ab5e4bd8af46498fbdf7dbdf91607ada, State: Initialized, Role: FOLLOWER
I20251027 14:57:32.783937 23247 consensus_queue.cc:260] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37079 } }
I20251027 14:57:32.784158 23247 raft_consensus.cc:399] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:32.784282 23247 raft_consensus.cc:493] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:32.784487 23247 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:32.786873 23247 raft_consensus.cc:515] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37079 } }
I20251027 14:57:32.787312 23247 leader_election.cc:304] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: ab5e4bd8af46498fbdf7dbdf91607ada; no voters:
I20251027 14:57:32.787847 23247 leader_election.cc:290] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:32.787962 23257 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:32.788606 23257 raft_consensus.cc:697] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [term 1 LEADER]: Becoming Leader. State: Replica: ab5e4bd8af46498fbdf7dbdf91607ada, State: Running, Role: LEADER
I20251027 14:57:32.789055 23257 consensus_queue.cc:237] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37079 } }
I20251027 14:57:32.789754 23247 sys_catalog.cc:565] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:32.791849 23258 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37079 } } }
I20251027 14:57:32.792166 23258 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:32.792923 23259 sys_catalog.cc:455] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [sys.catalog]: SysCatalogTable state changed. Reason: New leader ab5e4bd8af46498fbdf7dbdf91607ada. Latest consensus state: current_term: 1 leader_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "ab5e4bd8af46498fbdf7dbdf91607ada" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 37079 } } }
I20251027 14:57:32.793174 23259 sys_catalog.cc:458] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:32.793889 23262 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:32.799655 23262 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:32.809506 23262 catalog_manager.cc:1357] Generated new cluster ID: ec90a64a607842f493592cc0f79ba94f
I20251027 14:57:32.809625 23262 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:32.829583 23262 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:32.829963 23262 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:32.837996 23262 catalog_manager.cc:6022] T 00000000000000000000000000000000 P ab5e4bd8af46498fbdf7dbdf91607ada: Generated new TSK 0
I20251027 14:57:32.839102 23262 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:32.849092 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23180
W20251027 14:57:32.850417 23242 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:49333
W20251027 14:57:32.851181 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23180: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:32.851303 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23180
2025-10-27T14:57:32Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/14 (2688 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/15
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:32 dist-test-slave-hxxt krb5kdc[23278](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:32 dist-test-slave-hxxt krb5kdc[23278](info): set up 2 sockets
Oct 27 14:57:32 dist-test-slave-hxxt krb5kdc[23278](info): commencing operation
krb5kdc: starting...
W20251027 14:57:34.740530 416 mini_kdc.cc:121] Time spent starting KDC: real 1.856s user 0.000s sys 0.027s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:34 dist-test-slave-hxxt krb5kdc[23278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:35.032639 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34899
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:37649
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34899
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=required
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:35.358285 23294 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:35.358595 23294 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:35.358701 23294 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:35.366724 23294 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:35.366827 23294 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:35.366883 23294 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:35.366935 23294 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:35.366995 23294 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:35.377386 23294 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:37649
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34899
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34899
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:35.378909 23294 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:35.380209 23294 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:35.388099 23300 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:35.388159 23302 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:35.388137 23299 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:35.389189 23294 server_base.cc:1047] running on GCE node
I20251027 14:57:35.389917 23294 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:35.391325 23294 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:35.392539 23294 hybrid_clock.cc:648] HybridClock initialized: now 1761577055392473 us; error 38 us; skew 500 ppm
I20251027 14:57:35.392858 23294 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:35 dist-test-slave-hxxt krb5kdc[23278](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577055, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:35.395727 23294 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:35.397300 23294 webserver.cc:492] Webserver started at http://127.0.104.62:40385/ using document root <none> and password file <none>
I20251027 14:57:35.397979 23294 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:35.398075 23294 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:35.398406 23294 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:35.400942 23294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "9e23cc28a794442895434d586478fffb"
format_stamp: "Formatted at 2025-10-27 14:57:35 on dist-test-slave-hxxt"
server_key: "1bf9fbbf7b80ba8d452acbd152fb844c"
server_key_iv: "78697df24d324293e183f382480de5e6"
server_key_version: "encryptionkey@0"
I20251027 14:57:35.401674 23294 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "9e23cc28a794442895434d586478fffb"
format_stamp: "Formatted at 2025-10-27 14:57:35 on dist-test-slave-hxxt"
server_key: "1bf9fbbf7b80ba8d452acbd152fb844c"
server_key_iv: "78697df24d324293e183f382480de5e6"
server_key_version: "encryptionkey@0"
I20251027 14:57:35.406716 23294 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20251027 14:57:35.409883 23309 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:35.411151 23294 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20251027 14:57:35.411302 23294 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "9e23cc28a794442895434d586478fffb"
format_stamp: "Formatted at 2025-10-27 14:57:35 on dist-test-slave-hxxt"
server_key: "1bf9fbbf7b80ba8d452acbd152fb844c"
server_key_iv: "78697df24d324293e183f382480de5e6"
server_key_version: "encryptionkey@0"
I20251027 14:57:35.411494 23294 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:35.445978 23294 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:35.449937 23294 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:35.450181 23294 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:35.468755 23294 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34899
I20251027 14:57:35.468775 23360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34899 every 8 connection(s)
I20251027 14:57:35.470218 23294 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:35.474314 23361 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:35.476028 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23294
I20251027 14:57:35.476224 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_15.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:35.476787 416 external_mini_cluster.cc:1468] Setting key 31d3d19551aa90a76f00e1fb78d1ae66
I20251027 14:57:35.486325 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb: Bootstrap starting.
Oct 27 14:57:35 dist-test-slave-hxxt krb5kdc[23278](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577054, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:35.490196 23361 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:35.491266 23361 log.cc:826] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:35.494493 23361 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb: No bootstrap required, opened a new log
I20251027 14:57:35.497924 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:35.478546 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39133 (local address 127.0.104.62:34899)
1027 14:57:35.479310 (+ 764us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:35.479321 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:35.479365 (+ 44us) server_negotiation.cc:408] Connection header received
1027 14:57:35.480609 (+ 1244us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:35.480632 (+ 23us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:35.481050 (+ 418us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:35.481395 (+ 345us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:35.483397 (+ 2002us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:35.484849 (+ 1452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:35.486441 (+ 1592us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:35.486960 (+ 519us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:35.490856 (+ 3896us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:35.490918 (+ 62us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:35.490935 (+ 17us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:35.490970 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:35.494124 (+ 3154us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:35.494788 (+ 664us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:35.494795 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:35.494802 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:35.494909 (+ 107us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:35.495243 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:35.495247 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:35.495249 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:35.495700 (+ 451us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:35.496021 (+ 321us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:35.496515 (+ 494us) server_negotiation.cc:300] Negotiation successful
1027 14:57:35.496922 (+ 407us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":598,"thread_start_us":371,"threads_started":1}
I20251027 14:57:35.499948 23361 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23cc28a794442895434d586478fffb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34899 } }
I20251027 14:57:35.500324 23361 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:35.500455 23361 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9e23cc28a794442895434d586478fffb, State: Initialized, Role: FOLLOWER
I20251027 14:57:35.501046 23361 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23cc28a794442895434d586478fffb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34899 } }
I20251027 14:57:35.501243 23361 raft_consensus.cc:399] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:35.501374 23361 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:35.501530 23361 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:35.501608 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:35.503688 23361 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23cc28a794442895434d586478fffb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34899 } }
I20251027 14:57:35.504207 23361 leader_election.cc:304] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 9e23cc28a794442895434d586478fffb; no voters:
I20251027 14:57:35.504894 23361 leader_election.cc:290] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:35.504990 23370 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:35.508230 23361 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:35.511775 23370 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [term 1 LEADER]: Becoming Leader. State: Replica: 9e23cc28a794442895434d586478fffb, State: Running, Role: LEADER
I20251027 14:57:35.512419 23370 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23cc28a794442895434d586478fffb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34899 } }
I20251027 14:57:35.516036 23371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "9e23cc28a794442895434d586478fffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23cc28a794442895434d586478fffb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34899 } } }
I20251027 14:57:35.516407 23371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:35.517853 23372 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9e23cc28a794442895434d586478fffb. Latest consensus state: current_term: 1 leader_uuid: "9e23cc28a794442895434d586478fffb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "9e23cc28a794442895434d586478fffb" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34899 } } }
I20251027 14:57:35.518111 23372 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:35.519797 23364 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:35.505854 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:54385 (local address 127.0.104.62:34899)
1027 14:57:35.506027 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:35.506034 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:35.506505 (+ 471us) server_negotiation.cc:408] Connection header received
1027 14:57:35.506769 (+ 264us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:35.506817 (+ 48us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:35.506920 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:35.507005 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:35.508886 (+ 1881us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:35.509878 (+ 992us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:35.511105 (+ 1227us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:35.511593 (+ 488us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:35.513271 (+ 1678us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:35.513297 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:35.513300 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:35.513331 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:35.517044 (+ 3713us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:35.517784 (+ 740us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:35.517790 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:35.517793 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:35.517880 (+ 87us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:35.518248 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:35.518261 (+ 13us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:35.518264 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:35.518477 (+ 213us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:35.518701 (+ 224us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:35.519247 (+ 546us) server_negotiation.cc:300] Negotiation successful
1027 14:57:35.519473 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:57:35.521514 23378 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:35.526439 23378 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:35.534224 23378 catalog_manager.cc:1357] Generated new cluster ID: 8d346c1c2ee34e3aa0fde0b4c43b7b1e
I20251027 14:57:35.534336 23378 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:35.555522 23378 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:35.556774 23378 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:35.573127 23378 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 9e23cc28a794442895434d586478fffb: Generated new TSK 0
I20251027 14:57:35.574131 23378 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:35.587282 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23294
I20251027 14:57:35.588495 23356 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:35.682068 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23294
2025-10-27T14:57:35Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/15 (2830 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/16
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:35 dist-test-slave-hxxt krb5kdc[23393](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:35 dist-test-slave-hxxt krb5kdc[23393](info): set up 2 sockets
Oct 27 14:57:35 dist-test-slave-hxxt krb5kdc[23393](info): commencing operation
krb5kdc: starting...
W20251027 14:57:37.576339 416 mini_kdc.cc:121] Time spent starting KDC: real 1.861s user 0.000s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:37 dist-test-slave-hxxt krb5kdc[23393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577057, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:37.857184 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34949
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:34187
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:34949
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=required
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:38.179812 23409 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:38.180140 23409 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:38.180274 23409 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:38.188359 23409 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:38.188504 23409 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:38.188586 23409 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:38.188647 23409 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:38.188732 23409 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:38.199283 23409 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:34187
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:34949
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:34949
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=required
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:38.200949 23409 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:38.202329 23409 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:38.210409 23415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:38.210780 23417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:38.210969 23414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:38.211725 23409 server_base.cc:1047] running on GCE node
I20251027 14:57:38.212539 23409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:38.213920 23409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:38.215163 23409 hybrid_clock.cc:648] HybridClock initialized: now 1761577058215083 us; error 47 us; skew 500 ppm
I20251027 14:57:38.215514 23409 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:38 dist-test-slave-hxxt krb5kdc[23393](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577058, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:38.218474 23409 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:38.220026 23409 webserver.cc:492] Webserver started at http://127.0.104.62:39619/ using document root <none> and password file <none>
I20251027 14:57:38.220811 23409 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:38.220901 23409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:38.221239 23409 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:38.223845 23409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "8edda36742df490c8705a0520820d7fe"
format_stamp: "Formatted at 2025-10-27 14:57:38 on dist-test-slave-hxxt"
server_key: "7c33178ee47913eda802ed799d5d1dcf"
server_key_iv: "a65bcc4e65a5bd9dc4f9f59fbacdfd83"
server_key_version: "encryptionkey@0"
I20251027 14:57:38.224704 23409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "8edda36742df490c8705a0520820d7fe"
format_stamp: "Formatted at 2025-10-27 14:57:38 on dist-test-slave-hxxt"
server_key: "7c33178ee47913eda802ed799d5d1dcf"
server_key_iv: "a65bcc4e65a5bd9dc4f9f59fbacdfd83"
server_key_version: "encryptionkey@0"
I20251027 14:57:38.230175 23409 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20251027 14:57:38.233542 23424 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:38.234931 23409 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:57:38.235096 23409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "8edda36742df490c8705a0520820d7fe"
format_stamp: "Formatted at 2025-10-27 14:57:38 on dist-test-slave-hxxt"
server_key: "7c33178ee47913eda802ed799d5d1dcf"
server_key_iv: "a65bcc4e65a5bd9dc4f9f59fbacdfd83"
server_key_version: "encryptionkey@0"
I20251027 14:57:38.235318 23409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:38.255853 23409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:38.263293 23409 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:38.263542 23409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:38.282555 23409 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:34949
I20251027 14:57:38.282568 23475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:34949 every 8 connection(s)
I20251027 14:57:38.284030 23409 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:38.288434 23476 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:38.290584 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23409
I20251027 14:57:38.290764 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_16.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:38.291152 416 external_mini_cluster.cc:1468] Setting key 56193da4ce5339c78228c753b77737e5
I20251027 14:57:38.299978 23476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe: Bootstrap starting.
I20251027 14:57:38.303015 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:38.292912 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48465 (local address 127.0.104.62:34949)
1027 14:57:38.293713 (+ 801us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:38.293724 (+ 11us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:38.293769 (+ 45us) server_negotiation.cc:408] Connection header received
1027 14:57:38.294941 (+ 1172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:38.294962 (+ 21us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:38.295379 (+ 417us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:38.295735 (+ 356us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:38.296712 (+ 977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:38.298075 (+ 1363us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:38.299679 (+ 1604us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:38.300181 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:38.300685 (+ 504us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:38.300777 (+ 92us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:38.300793 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:38.301367 (+ 574us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:38.301424 (+ 57us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:38.301451 (+ 27us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:38.301549 (+ 98us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:38.301798 (+ 249us) server_negotiation.cc:300] Negotiation successful
1027 14:57:38.302066 (+ 268us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":618,"thread_start_us":385,"threads_started":1}
I20251027 14:57:38.303548 23476 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:38.304795 23476 log.cc:826] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:38.306907 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:38.307936 23476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe: No bootstrap required, opened a new log
I20251027 14:57:38.314800 23476 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edda36742df490c8705a0520820d7fe" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34949 } }
I20251027 14:57:38.315140 23476 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:38.315263 23476 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8edda36742df490c8705a0520820d7fe, State: Initialized, Role: FOLLOWER
I20251027 14:57:38.315893 23476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edda36742df490c8705a0520820d7fe" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34949 } }
I20251027 14:57:38.316159 23476 raft_consensus.cc:399] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:38.316325 23476 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:38.316538 23476 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:38.317454 23479 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:38.311794 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:35427 (local address 127.0.104.62:34949)
1027 14:57:38.311972 (+ 178us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:38.311978 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:38.312369 (+ 391us) server_negotiation.cc:408] Connection header received
1027 14:57:38.312550 (+ 181us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:38.312556 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:38.312632 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:38.312745 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:38.313418 (+ 673us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:38.314649 (+ 1231us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:38.315886 (+ 1237us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:38.316453 (+ 567us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:38.316730 (+ 277us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:38.316780 (+ 50us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:38.316784 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:38.316802 (+ 18us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:38.316841 (+ 39us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:38.316856 (+ 15us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:38.316930 (+ 74us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:38.317087 (+ 157us) server_negotiation.cc:300] Negotiation successful
1027 14:57:38.317209 (+ 122us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":51}
I20251027 14:57:38.319070 23476 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edda36742df490c8705a0520820d7fe" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34949 } }
I20251027 14:57:38.319552 23476 leader_election.cc:304] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 8edda36742df490c8705a0520820d7fe; no voters:
I20251027 14:57:38.320128 23476 leader_election.cc:290] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:38.320266 23486 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:38.320902 23486 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [term 1 LEADER]: Becoming Leader. State: Replica: 8edda36742df490c8705a0520820d7fe, State: Running, Role: LEADER
I20251027 14:57:38.321511 23486 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edda36742df490c8705a0520820d7fe" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34949 } }
I20251027 14:57:38.322335 23476 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:38.324527 23488 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8edda36742df490c8705a0520820d7fe. Latest consensus state: current_term: 1 leader_uuid: "8edda36742df490c8705a0520820d7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edda36742df490c8705a0520820d7fe" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34949 } } }
I20251027 14:57:38.324849 23488 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:38.324685 23487 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8edda36742df490c8705a0520820d7fe" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "8edda36742df490c8705a0520820d7fe" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 34949 } } }
I20251027 14:57:38.325237 23487 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:38.327346 23495 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:38.332500 23495 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:38.340688 23495 catalog_manager.cc:1357] Generated new cluster ID: a722aef027b5494692bf1259ff76808b
I20251027 14:57:38.340801 23495 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:38.368409 23495 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:38.369628 23495 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:38.380038 23495 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 8edda36742df490c8705a0520820d7fe: Generated new TSK 0
I20251027 14:57:38.380908 23495 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:38.394549 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23409
W20251027 14:57:38.395651 23471 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48465
W20251027 14:57:38.396356 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23409: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:38.396521 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23409
2025-10-27T14:57:38Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/16 (2716 ms)
[ RUN ] AuthTokenIssuingTest.ChannelConfidentiality/17
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:38 dist-test-slave-hxxt krb5kdc[23507](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:38 dist-test-slave-hxxt krb5kdc[23507](info): set up 2 sockets
Oct 27 14:57:38 dist-test-slave-hxxt krb5kdc[23507](info): commencing operation
krb5kdc: starting...
W20251027 14:57:40.292428 416 mini_kdc.cc:121] Time spent starting KDC: real 1.861s user 0.002s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:40 dist-test-slave-hxxt krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577060, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:40Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:40.581480 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42459
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:43685
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:42459
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--user_acl=*
--rpc_authentication=disabled
--rpc_encryption=disabled
--rpc_encrypt_loopback_connections=false with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:40.910188 23523 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:40.910498 23523 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:40.910636 23523 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:40.918540 23523 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:40.918642 23523 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:40.918728 23523 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:40.918787 23523 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:40.918872 23523 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:40.929031 23523 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:43685
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:42459
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:42459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=disabled
--rpc_encryption=disabled
--superuser_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:40.930632 23523 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:40.931963 23523 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:40.939529 23528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:40.939837 23529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:40.940239 23531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:40.940789 23523 server_base.cc:1047] running on GCE node
I20251027 14:57:40.941514 23523 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:40.942771 23523 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:40.944010 23523 hybrid_clock.cc:648] HybridClock initialized: now 1761577060943945 us; error 31 us; skew 500 ppm
I20251027 14:57:40.944320 23523 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:40 dist-test-slave-hxxt krb5kdc[23507](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577060, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:40.947233 23523 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:40.948712 23523 webserver.cc:492] Webserver started at http://127.0.104.62:44387/ using document root <none> and password file <none>
I20251027 14:57:40.949388 23523 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:40.949470 23523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:40.949771 23523 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:40.952265 23523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "94836930e3b348acb2f0de340d078b1d"
format_stamp: "Formatted at 2025-10-27 14:57:40 on dist-test-slave-hxxt"
server_key: "1e0fcd7fcf7aa6c2cd4bf1c9b79a4c9c"
server_key_iv: "2b6429e965383946d4232c1f81e60e8b"
server_key_version: "encryptionkey@0"
I20251027 14:57:40.953028 23523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "94836930e3b348acb2f0de340d078b1d"
format_stamp: "Formatted at 2025-10-27 14:57:40 on dist-test-slave-hxxt"
server_key: "1e0fcd7fcf7aa6c2cd4bf1c9b79a4c9c"
server_key_iv: "2b6429e965383946d4232c1f81e60e8b"
server_key_version: "encryptionkey@0"
I20251027 14:57:40.958101 23523 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:57:40.961303 23538 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:40.962538 23523 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20251027 14:57:40.962700 23523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "94836930e3b348acb2f0de340d078b1d"
format_stamp: "Formatted at 2025-10-27 14:57:40 on dist-test-slave-hxxt"
server_key: "1e0fcd7fcf7aa6c2cd4bf1c9b79a4c9c"
server_key_iv: "2b6429e965383946d4232c1f81e60e8b"
server_key_version: "encryptionkey@0"
I20251027 14:57:40.962890 23523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:40.968863 23523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:40.972800 23523 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:40.973044 23523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:40.991523 23523 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:42459
I20251027 14:57:40.991544 23589 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:42459 every 8 connection(s)
I20251027 14:57:40.993008 23523 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:40.994290 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23523
I20251027 14:57:40.994459 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.AuthTokenIssuingTest.ChannelConfidentiality_17.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:40.994818 416 external_mini_cluster.cc:1468] Setting key 3425e755e5508ce8e761dbe39db066b6
I20251027 14:57:40.997865 23590 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:41.003553 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:40.996514 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:48991 (local address 127.0.104.62:42459)
1027 14:57:40.997277 (+ 763us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:40.997302 (+ 25us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:40.997359 (+ 57us) server_negotiation.cc:408] Connection header received
1027 14:57:40.999195 (+ 1836us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:40.999224 (+ 29us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:40.999666 (+ 442us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:41.000163 (+ 497us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:41.001646 (+ 1483us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:41.001655 (+ 9us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:41.001671 (+ 16us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:41.001687 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:41.001764 (+ 77us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:41.001789 (+ 25us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:41.002002 (+ 213us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:41.002151 (+ 149us) server_negotiation.cc:300] Negotiation successful
1027 14:57:41.002357 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":562,"thread_start_us":330,"threads_started":1}
I20251027 14:57:41.007510 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:41.011343 23590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d: Bootstrap starting.
I20251027 14:57:41.014961 23593 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:41.012675 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:41061 (local address 127.0.104.62:42459)
1027 14:57:41.012848 (+ 173us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:41.012855 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:41.013223 (+ 368us) server_negotiation.cc:408] Connection header received
1027 14:57:41.013365 (+ 142us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:41.013372 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:41.013432 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:41.013539 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:41.014398 (+ 859us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:41.014404 (+ 6us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:41.014408 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
1027 14:57:41.014424 (+ 16us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:41.014472 (+ 48us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
1027 14:57:41.014491 (+ 19us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:41.014604 (+ 113us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:41.014646 (+ 42us) server_negotiation.cc:300] Negotiation successful
1027 14:57:41.014695 (+ 49us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:57:41.015164 23590 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:41.016476 23590 log.cc:826] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:41.019148 23590 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d: No bootstrap required, opened a new log
I20251027 14:57:41.024294 23590 raft_consensus.cc:359] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94836930e3b348acb2f0de340d078b1d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42459 } }
I20251027 14:57:41.024681 23590 raft_consensus.cc:385] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20251027 14:57:41.024778 23590 raft_consensus.cc:740] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 94836930e3b348acb2f0de340d078b1d, State: Initialized, Role: FOLLOWER
I20251027 14:57:41.025316 23590 consensus_queue.cc:260] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94836930e3b348acb2f0de340d078b1d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42459 } }
I20251027 14:57:41.025506 23590 raft_consensus.cc:399] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20251027 14:57:41.025625 23590 raft_consensus.cc:493] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20251027 14:57:41.025758 23590 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:41.027904 23590 raft_consensus.cc:515] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94836930e3b348acb2f0de340d078b1d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42459 } }
I20251027 14:57:41.028402 23590 leader_election.cc:304] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 94836930e3b348acb2f0de340d078b1d; no voters:
I20251027 14:57:41.028941 23590 leader_election.cc:290] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [CANDIDATE]: Term 1 election: Requested vote from peers
I20251027 14:57:41.029068 23600 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:41.029655 23600 raft_consensus.cc:697] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [term 1 LEADER]: Becoming Leader. State: Replica: 94836930e3b348acb2f0de340d078b1d, State: Running, Role: LEADER
I20251027 14:57:41.030161 23600 consensus_queue.cc:237] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94836930e3b348acb2f0de340d078b1d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42459 } }
I20251027 14:57:41.030838 23590 sys_catalog.cc:565] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:41.033185 23601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "94836930e3b348acb2f0de340d078b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94836930e3b348acb2f0de340d078b1d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42459 } } }
I20251027 14:57:41.033210 23602 sys_catalog.cc:455] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 94836930e3b348acb2f0de340d078b1d. Latest consensus state: current_term: 1 leader_uuid: "94836930e3b348acb2f0de340d078b1d" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "94836930e3b348acb2f0de340d078b1d" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 42459 } } }
I20251027 14:57:41.033532 23602 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:41.033540 23601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:41.034993 23607 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:41.040066 23607 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:41.048772 23607 catalog_manager.cc:1357] Generated new cluster ID: 70cf9f86477d44fd81826a6179382068
I20251027 14:57:41.048959 23607 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:41.075104 23607 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:41.075512 23607 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:41.084661 23607 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 94836930e3b348acb2f0de340d078b1d: Generated new TSK 0
I20251027 14:57:41.085525 23607 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:41.100503 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23523
W20251027 14:57:41.101830 23585 server_base.cc:1129] Unauthorized access attempt to method kudu.server.GenericService.CheckLeaks from {username='slave'} at 127.0.0.1:48991
W20251027 14:57:41.102510 416 external_mini_cluster.cc:1715] Unable to check leaks on /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23523: Remote error: Not authorized: unauthorized access to method: CheckLeaks
I20251027 14:57:41.102627 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23523
2025-10-27T14:57:41Z chronyd exiting
[ OK ] AuthTokenIssuingTest.ChannelConfidentiality/17 (2703 ms)
[----------] 18 tests from AuthTokenIssuingTest (67227 ms total)
[----------] 2 tests from ConnectToFollowerMasterTest
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:41 dist-test-slave-hxxt krb5kdc[23621](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:41 dist-test-slave-hxxt krb5kdc[23621](info): set up 2 sockets
Oct 27 14:57:41 dist-test-slave-hxxt krb5kdc[23621](info): commencing operation
krb5kdc: starting...
W20251027 14:57:42.988472 416 mini_kdc.cc:121] Time spent starting KDC: real 1.853s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:43 dist-test-slave-hxxt krb5kdc[23621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:43Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:43.286284 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38221
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38221,127.0.104.61:40605,127.0.104.60:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:43.607169 23637 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251027 14:57:43.607518 23637 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:43.607583 23637 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:43.607690 23637 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:43.615985 23637 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:43.616117 23637 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:43.616166 23637 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:43.616233 23637 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:43.616282 23637 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:43.627197 23637 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38221,127.0.104.61:40605,127.0.104.60:43317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:38221
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:43.628789 23637 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:43.630126 23637 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:43.638108 23643 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:43.638134 23642 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:43.638803 23645 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:43.639096 23637 server_base.cc:1047] running on GCE node
I20251027 14:57:43.639814 23637 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:43.641263 23637 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:43.642452 23637 hybrid_clock.cc:648] HybridClock initialized: now 1761577063642394 us; error 54 us; skew 500 ppm
I20251027 14:57:43.642776 23637 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:43 dist-test-slave-hxxt krb5kdc[23621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:43.645812 23637 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:43.647477 23637 webserver.cc:492] Webserver started at http://127.0.104.62:33215/ using document root <none> and password file <none>
I20251027 14:57:43.648427 23637 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:43.648547 23637 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:43.648880 23637 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:43.651554 23637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e"
format_stamp: "Formatted at 2025-10-27 14:57:43 on dist-test-slave-hxxt"
server_key: "1f7c42f4c240b7bcab55fcf9390bab1c"
server_key_iv: "6513619eddec2b384d495c9882f7a16e"
server_key_version: "encryptionkey@0"
I20251027 14:57:43.652370 23637 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e"
format_stamp: "Formatted at 2025-10-27 14:57:43 on dist-test-slave-hxxt"
server_key: "1f7c42f4c240b7bcab55fcf9390bab1c"
server_key_iv: "6513619eddec2b384d495c9882f7a16e"
server_key_version: "encryptionkey@0"
I20251027 14:57:43.657990 23637 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20251027 14:57:43.661702 23652 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:43.663141 23637 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:57:43.663340 23637 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e"
format_stamp: "Formatted at 2025-10-27 14:57:43 on dist-test-slave-hxxt"
server_key: "1f7c42f4c240b7bcab55fcf9390bab1c"
server_key_iv: "6513619eddec2b384d495c9882f7a16e"
server_key_version: "encryptionkey@0"
I20251027 14:57:43.663538 23637 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:43.691509 23637 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:43.698733 23637 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:43.699010 23637 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:43.717180 23637 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:38221
I20251027 14:57:43.717206 23703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:38221 every 8 connection(s)
I20251027 14:57:43.718619 23637 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:43.719697 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23637
I20251027 14:57:43.719929 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:43.720705 416 external_mini_cluster.cc:1468] Setting key 355668dee86a9d96817fd6d313218136
I20251027 14:57:43.723798 23704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:43.732141 23704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:43 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.61@KRBTEST.COM" created.
I20251027 14:57:43.752823 23707 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.735417 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53820 (local address 127.0.104.62:38221)
1027 14:57:43.736233 (+ 816us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:43.736249 (+ 16us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:43.736730 (+ 481us) server_negotiation.cc:408] Connection header received
1027 14:57:43.738272 (+ 1542us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:43.738279 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:43.738528 (+ 249us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:43.738716 (+ 188us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:43.740205 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:43.741429 (+ 1224us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:43.742414 (+ 985us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:43.742916 (+ 502us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:43.745978 (+ 3062us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:43.746042 (+ 64us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:43.746056 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:43.746083 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:43.748854 (+ 2771us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:43.749579 (+ 725us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:43.749587 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:43.749589 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:43.749686 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:43.750054 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:43.750057 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:43.750059 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:43.750546 (+ 487us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:43.750694 (+ 148us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:43.751307 (+ 613us) server_negotiation.cc:300] Negotiation successful
1027 14:57:43.751596 (+ 289us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":608,"thread_start_us":409,"threads_started":1}
I20251027 14:57:43.752846 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.735417 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38221 (local address 127.0.0.1:53820)
1027 14:57:43.736556 (+ 1139us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.736598 (+ 42us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:43.737879 (+ 1281us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:43.738754 (+ 875us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:43.738761 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:43.739517 (+ 756us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:43.740001 (+ 484us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:43.740020 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:43.741556 (+ 1536us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:43.741561 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:43.742287 (+ 726us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:43.742298 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:43.742573 (+ 275us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:43.743424 (+ 851us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:43.743458 (+ 34us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:43.745815 (+ 2357us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:43.748972 (+ 3157us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:43.748983 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:43.748996 (+ 13us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:43.749436 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:43.749789 (+ 353us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:43.749794 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:43.749798 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:43.749963 (+ 165us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:43.750677 (+ 714us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:43.750686 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:43.750907 (+ 221us) client_negotiation.cc:769] Sending connection context
1027 14:57:43.751235 (+ 328us) client_negotiation.cc:240] Negotiation successful
1027 14:57:43.751782 (+ 547us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":923,"mutex_wait_us":101,"thread_start_us":750,"threads_started":1}
I20251027 14:57:43.755717 23704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:43.756754 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.756339 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33460)
1027 14:57:43.756511 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.756615 (+ 104us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":68}
W20251027 14:57:43.756995 23656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
I20251027 14:57:43.758384 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.758111 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33472)
1027 14:57:43.758223 (+ 112us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.758282 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20251027 14:57:43.758579 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40605: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.61@KRBTEST.COM" created.
I20251027 14:57:43.807868 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } attempt: 1
I20251027 14:57:43.808897 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.808500 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33488)
1027 14:57:43.808671 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.808768 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
I20251027 14:57:43.809800 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.809495 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33496)
1027 14:57:43.809618 (+ 123us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.809678 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251027 14:57:43.810021 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40605: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:43.824996 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40605
--webserver_interface=127.0.104.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38221,127.0.104.61:40605,127.0.104.60:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:57:43.878258 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } attempt: 2
I20251027 14:57:43.879313 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.878860 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33504)
1027 14:57:43.879075 (+ 215us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.879178 (+ 103us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20251027 14:57:43.880234 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.879896 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33512)
1027 14:57:43.880030 (+ 134us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.880098 (+ 68us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:57:43.880414 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40605: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
I20251027 14:57:43.971657 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } attempt: 3
I20251027 14:57:43.972677 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.972237 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33520)
1027 14:57:43.972453 (+ 216us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.972544 (+ 91us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":71}
I20251027 14:57:43.973621 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:43.973276 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33534)
1027 14:57:43.973420 (+ 144us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:43.973493 (+ 73us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251027 14:57:43.973798 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40605: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
I20251027 14:57:44.117028 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } attempt: 4
I20251027 14:57:44.118260 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.117724 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33536)
1027 14:57:44.118000 (+ 276us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.118085 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":158}
I20251027 14:57:44.119199 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.118855 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33544)
1027 14:57:44.118987 (+ 132us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.119061 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:57:44.119380 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40605: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40605: connect: Connection refused (error 111)
W20251027 14:57:44.149657 23713 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251027 14:57:44.150033 23713 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:44.150116 23713 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:44.150255 23713 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:44.158262 23713 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:44.158388 23713 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:44.158466 23713 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:44.158548 23713 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:44.158605 23713 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:44.169322 23713 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38221,127.0.104.61:40605,127.0.104.60:43317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40605
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:44.170900 23713 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:44.172196 23713 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:44.179798 23719 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:44.179852 23718 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:44.180305 23721 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:44.180706 23713 server_base.cc:1047] running on GCE node
I20251027 14:57:44.181423 23713 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:44.182762 23713 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:44.183970 23713 hybrid_clock.cc:648] HybridClock initialized: now 1761577064183916 us; error 31 us; skew 500 ppm
I20251027 14:57:44.184310 23713 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:44.187337 23713 init.cc:377] Logged in from keytab as kudu/127.0.104.61@KRBTEST.COM (short username kudu)
I20251027 14:57:44.188966 23713 webserver.cc:492] Webserver started at http://127.0.104.61:40479/ using document root <none> and password file <none>
I20251027 14:57:44.189661 23713 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:44.189759 23713 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:44.190114 23713 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:44.192787 23713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data/instance:
uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2"
format_stamp: "Formatted at 2025-10-27 14:57:44 on dist-test-slave-hxxt"
server_key: "21f205c48a70ed0231d0426eb6e8fd02"
server_key_iv: "30b69cf82125cf2eda315bb784fc28ca"
server_key_version: "encryptionkey@0"
I20251027 14:57:44.193537 23713 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal/instance:
uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2"
format_stamp: "Formatted at 2025-10-27 14:57:44 on dist-test-slave-hxxt"
server_key: "21f205c48a70ed0231d0426eb6e8fd02"
server_key_iv: "30b69cf82125cf2eda315bb784fc28ca"
server_key_version: "encryptionkey@0"
I20251027 14:57:44.198808 23713 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20251027 14:57:44.202312 23728 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:44.203644 23713 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20251027 14:57:44.203805 23713 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal
uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2"
format_stamp: "Formatted at 2025-10-27 14:57:44 on dist-test-slave-hxxt"
server_key: "21f205c48a70ed0231d0426eb6e8fd02"
server_key_iv: "30b69cf82125cf2eda315bb784fc28ca"
server_key_version: "encryptionkey@0"
I20251027 14:57:44.203989 23713 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:44.218891 23713 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:44.223284 23713 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:44.223562 23713 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:44.242472 23779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40605 every 8 connection(s)
I20251027 14:57:44.242468 23713 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40605
I20251027 14:57:44.244055 23713 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
I20251027 14:57:44.248222 23780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:44.248453 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23713
I20251027 14:57:44.248654 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-1/wal/instance
I20251027 14:57:44.249166 416 external_mini_cluster.cc:1468] Setting key 0bd82feea05ac7281bfa68449cc2d728
I20251027 14:57:44.256742 23780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:44.277802 23782 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.259651 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53826 (local address 127.0.104.62:38221)
1027 14:57:44.260194 (+ 543us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.260200 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.261247 (+ 1047us) server_negotiation.cc:408] Connection header received
1027 14:57:44.262982 (+ 1735us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.262987 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.263058 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.263168 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.265415 (+ 2247us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.266333 (+ 918us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.267602 (+ 1269us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.267917 (+ 315us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.271203 (+ 3286us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.271244 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.271249 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.271287 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.274591 (+ 3304us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.275358 (+ 767us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.275365 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.275369 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.275470 (+ 101us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.275804 (+ 334us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.275809 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.275813 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.276193 (+ 380us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.276344 (+ 151us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.277108 (+ 764us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.277372 (+ 264us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":421,"thread_start_us":283,"threads_started":1}
WARNING: no policy specified for kudu/127.0.104.60@KRBTEST.COM; defaulting to no policy
I20251027 14:57:44.278414 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.260238 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38221 (local address 127.0.0.1:53826)
1027 14:57:44.261062 (+ 824us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.261109 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.262669 (+ 1560us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.263421 (+ 752us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.263440 (+ 19us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.264238 (+ 798us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.265184 (+ 946us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.265210 (+ 26us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.266479 (+ 1269us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.266484 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.267481 (+ 997us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.267494 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.267759 (+ 265us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.268491 (+ 732us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.268522 (+ 31us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.270974 (+ 2452us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.274750 (+ 3776us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.274757 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.274771 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.275213 (+ 442us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.275598 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.275601 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.275603 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.275703 (+ 100us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.276315 (+ 612us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.276322 (+ 7us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.276678 (+ 356us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.276949 (+ 271us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.277310 (+ 361us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":570,"thread_start_us":362,"threads_started":1}
I20251027 14:57:44.280120 23780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.60@KRBTEST.COM" created.
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:44.293852 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.280959 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33546)
1027 14:57:44.281131 (+ 172us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.281157 (+ 26us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.281267 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.282337 (+ 1070us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.282341 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.282656 (+ 315us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.282954 (+ 298us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.282962 (+ 8us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.284866 (+ 1904us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.284870 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.285586 (+ 716us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.285597 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.285704 (+ 107us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.286308 (+ 604us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.286331 (+ 23us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.288396 (+ 2065us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.291841 (+ 3445us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.291847 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.291851 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.292167 (+ 316us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.292484 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.292487 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.292490 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.292557 (+ 67us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.293173 (+ 616us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.293177 (+ 4us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.293291 (+ 114us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.293418 (+ 127us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.293624 (+ 206us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20251027 14:57:44.294224 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.281092 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33546 (local address 127.0.104.61:40605)
1027 14:57:44.281799 (+ 707us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.281808 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.281837 (+ 29us) server_negotiation.cc:408] Connection header received
1027 14:57:44.281886 (+ 49us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.281894 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.282153 (+ 259us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.282319 (+ 166us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.283158 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.284742 (+ 1584us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.285716 (+ 974us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.286177 (+ 461us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.288537 (+ 2360us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.288593 (+ 56us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.288606 (+ 13us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.288632 (+ 26us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.291714 (+ 3082us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.292289 (+ 575us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.292297 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.292300 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.292395 (+ 95us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.292637 (+ 242us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.292641 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.292644 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.293020 (+ 376us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.293191 (+ 171us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.293540 (+ 349us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.293819 (+ 279us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":574,"thread_start_us":428,"threads_started":1}
I20251027 14:57:44.296413 23780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:44.297427 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.297012 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38714)
1027 14:57:44.297170 (+ 158us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.297281 (+ 111us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251027 14:57:44.297678 23729 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
I20251027 14:57:44.299201 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.298852 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38720)
1027 14:57:44.299000 (+ 148us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.299071 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":53}
W20251027 14:57:44.299423 23780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.60@KRBTEST.COM" created.
I20251027 14:57:44.348722 23780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 1
I20251027 14:57:44.349887 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.349310 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38732)
1027 14:57:44.349564 (+ 254us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.349661 (+ 97us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":65}
I20251027 14:57:44.350927 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.350561 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38734)
1027 14:57:44.350702 (+ 141us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.350779 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":44}
W20251027 14:57:44.351116 23780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:44.356819 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43317
--webserver_interface=127.0.104.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:39221
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:38221,127.0.104.61:40605,127.0.104.60:43317
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=optional with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:57:44.418620 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } attempt: 5
I20251027 14:57:44.419327 23780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 2
I20251027 14:57:44.420318 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.419841 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38738)
1027 14:57:44.419992 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.420105 (+ 113us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
I20251027 14:57:44.421231 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.420934 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38752)
1027 14:57:44.421055 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.421119 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251027 14:57:44.421504 23780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:44.431735 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.419338 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33548)
1027 14:57:44.419604 (+ 266us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.419622 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.419761 (+ 139us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.420173 (+ 412us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.420179 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.420693 (+ 514us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.421193 (+ 500us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.421209 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.422781 (+ 1572us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.422785 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.423506 (+ 721us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.423518 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.423631 (+ 113us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.424261 (+ 630us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.424286 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.426165 (+ 1879us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.429836 (+ 3671us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.429840 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.429843 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.430169 (+ 326us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.430480 (+ 311us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.430484 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.430486 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.430551 (+ 65us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.431016 (+ 465us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.431019 (+ 3us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.431169 (+ 150us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.431276 (+ 107us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.431484 (+ 208us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":160}
I20251027 14:57:44.431927 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.419411 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.104.61:40605)
1027 14:57:44.419579 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.419585 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.419718 (+ 133us) server_negotiation.cc:408] Connection header received
1027 14:57:44.419881 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.419887 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.419996 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.420091 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.421381 (+ 1290us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.422646 (+ 1265us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.423642 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.424110 (+ 468us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.426329 (+ 2219us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.426365 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.426370 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.426405 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.429695 (+ 3290us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.430280 (+ 585us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.430286 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.430289 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.430391 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.430629 (+ 238us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.430634 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.430637 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.430925 (+ 288us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.431087 (+ 162us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.431397 (+ 310us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.431639 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:57:44.432888 23704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:44.433902 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.433541 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38766)
1027 14:57:44.433692 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.433757 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":50}
I20251027 14:57:44.434820 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.434538 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38776)
1027 14:57:44.434674 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.434733 (+ 59us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251027 14:57:44.435015 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
I20251027 14:57:44.486195 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 1
I20251027 14:57:44.487263 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.486827 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38790)
1027 14:57:44.487020 (+ 193us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.487103 (+ 83us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":81}
I20251027 14:57:44.488181 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.487893 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38798)
1027 14:57:44.488008 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.488068 (+ 60us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251027 14:57:44.488394 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
I20251027 14:57:44.512696 23780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 3
I20251027 14:57:44.513707 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.513272 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38812)
1027 14:57:44.513480 (+ 208us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.513558 (+ 78us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20251027 14:57:44.514705 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.514325 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38828)
1027 14:57:44.514528 (+ 203us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.514599 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":113}
W20251027 14:57:44.514930 23780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
I20251027 14:57:44.556581 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 2
I20251027 14:57:44.557636 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.557189 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38838)
1027 14:57:44.557385 (+ 196us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.557470 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":62}
I20251027 14:57:44.558636 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.558340 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38848)
1027 14:57:44.558467 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.558533 (+ 66us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
W20251027 14:57:44.558854 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
I20251027 14:57:44.658128 23780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 4
I20251027 14:57:44.659296 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.658831 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38854)
1027 14:57:44.659048 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.659138 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":89}
I20251027 14:57:44.660262 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.659966 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38866)
1027 14:57:44.660087 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.660152 (+ 65us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":42}
W20251027 14:57:44.660497 23780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
I20251027 14:57:44.665062 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 3
I20251027 14:57:44.665990 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.665606 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38868)
1027 14:57:44.665760 (+ 154us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.665834 (+ 74us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":55}
I20251027 14:57:44.667011 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.666587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38880)
1027 14:57:44.666789 (+ 202us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.666888 (+ 99us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":114}
W20251027 14:57:44.667229 23704 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:43317: Network error: Client connection negotiation failed: client connection to 127.0.104.60:43317: connect: Connection refused (error 111)
W20251027 14:57:44.686082 23790 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251027 14:57:44.686412 23790 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:44.686473 23790 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:44.686579 23790 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:44.694945 23790 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:44.695068 23790 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:44.695116 23790 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:44.695178 23790 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:44.695223 23790 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:44.705976 23790 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:39221
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:38221,127.0.104.61:40605,127.0.104.60:43317
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:43317
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:44.707511 23790 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:44.708873 23790 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:44.716799 23796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:44.716820 23798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:44.717279 23795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:44.718014 23790 server_base.cc:1047] running on GCE node
I20251027 14:57:44.718705 23790 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:44.720057 23790 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:44.721246 23790 hybrid_clock.cc:648] HybridClock initialized: now 1761577064721184 us; error 57 us; skew 500 ppm
I20251027 14:57:44.721562 23790 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:44.724587 23790 init.cc:377] Logged in from keytab as kudu/127.0.104.60@KRBTEST.COM (short username kudu)
I20251027 14:57:44.726109 23790 webserver.cc:492] Webserver started at http://127.0.104.60:39135/ using document root <none> and password file <none>
I20251027 14:57:44.726779 23790 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:44.726868 23790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:44.727187 23790 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:44.729842 23790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data/instance:
uuid: "29ad36b2fdbc4f058ea6a2209708b12f"
format_stamp: "Formatted at 2025-10-27 14:57:44 on dist-test-slave-hxxt"
server_key: "efeceef9c5c0bac49bada9188b9388fc"
server_key_iv: "60eb0ca5e761c0d4254735ba11228918"
server_key_version: "encryptionkey@0"
I20251027 14:57:44.730592 23790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal/instance:
uuid: "29ad36b2fdbc4f058ea6a2209708b12f"
format_stamp: "Formatted at 2025-10-27 14:57:44 on dist-test-slave-hxxt"
server_key: "efeceef9c5c0bac49bada9188b9388fc"
server_key_iv: "60eb0ca5e761c0d4254735ba11228918"
server_key_version: "encryptionkey@0"
I20251027 14:57:44.735687 23790 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20251027 14:57:44.738932 23805 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:44.740139 23790 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.000s sys 0.002s
I20251027 14:57:44.740283 23790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal
uuid: "29ad36b2fdbc4f058ea6a2209708b12f"
format_stamp: "Formatted at 2025-10-27 14:57:44 on dist-test-slave-hxxt"
server_key: "efeceef9c5c0bac49bada9188b9388fc"
server_key_iv: "60eb0ca5e761c0d4254735ba11228918"
server_key_version: "encryptionkey@0"
I20251027 14:57:44.740490 23790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:44.754565 23790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:44.758164 23790 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:44.758415 23790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:44.776542 23790 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:43317
I20251027 14:57:44.776561 23856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:43317 every 8 connection(s)
I20251027 14:57:44.777981 23790 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
I20251027 14:57:44.780030 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 23790
I20251027 14:57:44.780220 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_0.1761576633729408-416-0/minicluster-data/master-2/wal/instance
I20251027 14:57:44.780694 416 external_mini_cluster.cc:1468] Setting key c5c6c4d3efea90eeb1878332a1b9a2d6
I20251027 14:57:44.782809 23857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:44.790309 23857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:44.801973 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.782442 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38894 (local address 127.0.104.60:43317)
1027 14:57:44.783167 (+ 725us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.783182 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.783238 (+ 56us) server_negotiation.cc:408] Connection header received
1027 14:57:44.785022 (+ 1784us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.785054 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.785532 (+ 478us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.786015 (+ 483us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.787491 (+ 1476us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.789235 (+ 1744us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.790437 (+ 1202us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.790958 (+ 521us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.794519 (+ 3561us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.794609 (+ 90us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.794623 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.794661 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.797882 (+ 3221us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.798636 (+ 754us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.798645 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.798654 (+ 9us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.798791 (+ 137us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.799173 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.799178 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.799185 (+ 7us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.799702 (+ 517us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.799951 (+ 249us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.800564 (+ 613us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.800995 (+ 431us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":532,"thread_start_us":309,"threads_started":1}
I20251027 14:57:44.805821 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.792616 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:38221 (local address 127.0.0.1:53828)
1027 14:57:44.793335 (+ 719us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.793374 (+ 39us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.793529 (+ 155us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.793934 (+ 405us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.793943 (+ 9us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.794534 (+ 591us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.795025 (+ 491us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.795035 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.796619 (+ 1584us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.796624 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.797503 (+ 879us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.797515 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.797672 (+ 157us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.798313 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.798338 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.800702 (+ 2364us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.803546 (+ 2844us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.803557 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.803564 (+ 7us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.803892 (+ 328us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.804189 (+ 297us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.804193 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.804195 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.804344 (+ 149us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.804857 (+ 513us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.804866 (+ 9us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.805050 (+ 184us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.805316 (+ 266us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.805560 (+ 244us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":566,"thread_start_us":446,"threads_started":1}
I20251027 14:57:44.806712 23861 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.792597 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53828 (local address 127.0.104.62:38221)
1027 14:57:44.793093 (+ 496us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.793099 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.793470 (+ 371us) server_negotiation.cc:408] Connection header received
1027 14:57:44.793664 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.793670 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.793746 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.793842 (+ 96us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.795194 (+ 1352us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.796483 (+ 1289us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.797657 (+ 1174us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.798112 (+ 455us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.800885 (+ 2773us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.800940 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.800944 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.800976 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.803426 (+ 2450us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.804000 (+ 574us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.804005 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.804007 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.804083 (+ 76us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.804465 (+ 382us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.804471 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.804475 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.804713 (+ 238us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.804862 (+ 149us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.806062 (+ 1200us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.806312 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":381,"thread_start_us":262,"threads_started":1}
I20251027 14:57:44.808037 23857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:44.818534 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.805011 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53832 (local address 127.0.104.62:38221)
1027 14:57:44.805477 (+ 466us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.805481 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.805497 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:57:44.805532 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.805535 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.805594 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.805678 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.807331 (+ 1653us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.808472 (+ 1141us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.809636 (+ 1164us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.810037 (+ 401us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.813450 (+ 3413us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.813508 (+ 58us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.813512 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.813539 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.816090 (+ 2551us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.816881 (+ 791us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.816885 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.816887 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.816951 (+ 64us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.817280 (+ 329us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.817283 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.817285 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.817494 (+ 209us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.817702 (+ 208us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.818025 (+ 323us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.818219 (+ 194us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":383,"thread_start_us":244,"threads_started":1}
I20251027 14:57:44.820022 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.808713 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40605 (local address 127.0.0.1:33564)
1027 14:57:44.808922 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.808938 (+ 16us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.809034 (+ 96us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.809321 (+ 287us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.809326 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.809625 (+ 299us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.809955 (+ 330us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.809967 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.811052 (+ 1085us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.811056 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.811734 (+ 678us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.811743 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.811888 (+ 145us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.812555 (+ 667us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.812577 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.814507 (+ 1930us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.817717 (+ 3210us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.817723 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.817727 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.818171 (+ 444us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.818545 (+ 374us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.818548 (+ 3us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.818550 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.818632 (+ 82us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.819210 (+ 578us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.819215 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.819399 (+ 184us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.819553 (+ 154us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.819794 (+ 241us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":130}
I20251027 14:57:44.820289 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.808751 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33564 (local address 127.0.104.61:40605)
1027 14:57:44.808922 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.808928 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.808984 (+ 56us) server_negotiation.cc:408] Connection header received
1027 14:57:44.809145 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.809149 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.809211 (+ 62us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.809306 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.810147 (+ 841us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.810917 (+ 770us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.811878 (+ 961us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.812298 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.814644 (+ 2346us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.814678 (+ 34us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.814681 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.814711 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.817564 (+ 2853us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.818307 (+ 743us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.818313 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.818316 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.818413 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.818776 (+ 363us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.818782 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.818785 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.819098 (+ 313us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.819212 (+ 114us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.819687 (+ 475us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.819927 (+ 240us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":35}
I20251027 14:57:44.821136 23857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:44.832767 23862 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.821701 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38900)
1027 14:57:44.821841 (+ 140us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.821859 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.821969 (+ 110us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.822247 (+ 278us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.822253 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.822650 (+ 397us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.823077 (+ 427us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.823087 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.824136 (+ 1049us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.824140 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.824854 (+ 714us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.824865 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.825014 (+ 149us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.825685 (+ 671us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.825702 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.827586 (+ 1884us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.830775 (+ 3189us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.830781 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.830785 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.831094 (+ 309us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.831337 (+ 243us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.831341 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.831344 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.831404 (+ 60us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.831801 (+ 397us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.831806 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.831926 (+ 120us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.832043 (+ 117us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.832308 (+ 265us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":45}
I20251027 14:57:44.832788 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.821740 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38900 (local address 127.0.104.60:43317)
1027 14:57:44.821867 (+ 127us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.821871 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.821922 (+ 51us) server_negotiation.cc:408] Connection header received
1027 14:57:44.822069 (+ 147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.822073 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.822125 (+ 52us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.822199 (+ 74us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.823218 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.824013 (+ 795us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.824973 (+ 960us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.825401 (+ 428us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.827720 (+ 2319us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.827744 (+ 24us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.827748 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.827772 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.830666 (+ 2894us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.831189 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.831194 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.831196 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.831253 (+ 57us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.831483 (+ 230us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.831488 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.831492 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.831704 (+ 212us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.831832 (+ 128us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.832133 (+ 301us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.832462 (+ 329us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":92,"server-negotiator.queue_time_us":48}
I20251027 14:57:44.835266 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.820660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33568 (local address 127.0.104.61:40605)
1027 14:57:44.820828 (+ 168us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.820833 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.820847 (+ 14us) server_negotiation.cc:408] Connection header received
1027 14:57:44.820894 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.820899 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.820966 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.821045 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.824161 (+ 3116us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.825020 (+ 859us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.826199 (+ 1179us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.826592 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.830247 (+ 3655us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.830276 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.830280 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.830311 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.832787 (+ 2476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.833548 (+ 761us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.833554 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.833557 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.833645 (+ 88us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.833960 (+ 315us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.833964 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.833967 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.834246 (+ 279us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.834534 (+ 288us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.834707 (+ 173us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.834941 (+ 234us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":38,"server-negotiator.queue_time_us":80}
I20251027 14:57:44.838534 23857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: Bootstrap starting.
I20251027 14:57:44.839985 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:44.841734 23857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:44.842998 23857 log.cc:826] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:44.844441 23704 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 4
I20251027 14:57:44.842654 23679 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53832
I20251027 14:57:44.846463 23857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: No bootstrap required, opened a new log
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577063, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
I20251027 14:57:44.852509 23857 raft_consensus.cc:359] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:44.852944 23857 raft_consensus.cc:740] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 29ad36b2fdbc4f058ea6a2209708b12f, State: Initialized, Role: FOLLOWER
I20251027 14:57:44.853431 23857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:44.854502 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:44.854797 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:44.855738 23857 sys_catalog.cc:565] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:44.859750 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.845102 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38910 (local address 127.0.104.60:43317)
1027 14:57:44.845271 (+ 169us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.845276 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.845296 (+ 20us) server_negotiation.cc:408] Connection header received
1027 14:57:44.845400 (+ 104us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.845405 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.845476 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.845584 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.846661 (+ 1077us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.847942 (+ 1281us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.849357 (+ 1415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.849810 (+ 453us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.852894 (+ 3084us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.852940 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.852945 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.852976 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.857047 (+ 4071us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.857833 (+ 786us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.857842 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.857846 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.857948 (+ 102us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.858316 (+ 368us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.858321 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.858323 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.858586 (+ 263us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.858712 (+ 126us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.859127 (+ 415us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.859388 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":31}
I20251027 14:57:44.859762 23708 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.844936 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38910)
1027 14:57:44.845154 (+ 218us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.845169 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.845300 (+ 131us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.845604 (+ 304us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.845609 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.846073 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.846528 (+ 455us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.846540 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.848071 (+ 1531us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.848076 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.849224 (+ 1148us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.849240 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.849374 (+ 134us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.850013 (+ 639us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.850038 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.852717 (+ 2679us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.857211 (+ 4494us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.857217 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.857222 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.857662 (+ 440us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.858081 (+ 419us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.858087 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.858090 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.858202 (+ 112us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.858690 (+ 488us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.858695 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.858843 (+ 148us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.859017 (+ 174us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.859344 (+ 327us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":120}
I20251027 14:57:44.866547 23704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e: Bootstrap starting.
W20251027 14:57:44.865502 23876 catalog_manager.cc:1568] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20251027 14:57:44.866981 23876 catalog_manager.cc:883] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20251027 14:57:44.870754 23704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:44.872105 23704 log.cc:826] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:44.869059 23790 master_runner.cc:186] Error getting master registration for 127.0.104.62:38221: OK, instance_id { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" instance_seqno: 1761577063712101 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 14:57:44.875685 23704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e: No bootstrap required, opened a new log
I20251027 14:57:44.875717 23790 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251027 14:57:44.881022 23704 raft_consensus.cc:359] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:44.881358 23704 raft_consensus.cc:740] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d5ac7e7dff3b4d6da6c515ffe19aa29e, State: Initialized, Role: FOLLOWER
I20251027 14:57:44.881940 23704 consensus_queue.cc:260] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:44.882867 23879 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:44.883167 23879 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:44.884150 23704 sys_catalog.cc:565] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:44.884207 23679 raft_consensus.cc:493] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 14:57:44.884442 23679 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:44.887276 23679 raft_consensus.cc:515] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:44.889039 23679 leader_election.cc:290] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [CANDIDATE]: Term 1 election: Requested vote from peers 49bf73c91c3e4e53a9d428cdae6b5ed2 (127.0.104.61:40605), 29ad36b2fdbc4f058ea6a2209708b12f (127.0.104.60:43317)
I20251027 14:57:44.889393 23832 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "29ad36b2fdbc4f058ea6a2209708b12f"
I20251027 14:57:44.889729 23832 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:44.892208 23832 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ac7e7dff3b4d6da6c515ffe19aa29e in term 1.
I20251027 14:57:44.892932 23653 leader_election.cc:304] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 29ad36b2fdbc4f058ea6a2209708b12f, d5ac7e7dff3b4d6da6c515ffe19aa29e; no voters:
I20251027 14:57:44.893400 23879 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:44.895241 23879 raft_consensus.cc:697] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [term 1 LEADER]: Becoming Leader. State: Replica: d5ac7e7dff3b4d6da6c515ffe19aa29e, State: Running, Role: LEADER
I20251027 14:57:44.895743 23879 consensus_queue.cc:237] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:44.909243 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ac7e7dff3b4d6da6c515ffe19aa29e. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:44.909660 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:44.919380 23891 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:44.932219 23891 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:44.938951 23900 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251027 14:57:44.943543 23637 master_runner.cc:186] Error getting master registration for 127.0.104.61:40605: OK, instance_id { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" instance_seqno: 1761577064237398 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 14:57:44.950166 23637 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251027 14:57:44.951725 23658 tablet.cc:2378] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:57:44.959700 23780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } attempt: 5
I20251027 14:57:44.963400 23863 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.928476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53838 (local address 127.0.104.62:38221)
1027 14:57:44.928638 (+ 162us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.928644 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.935955 (+ 7311us) server_negotiation.cc:408] Connection header received
1027 14:57:44.936250 (+ 295us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.936257 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.936347 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.936456 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.948520 (+ 12064us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.949885 (+ 1365us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.952571 (+ 2686us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.953085 (+ 514us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.954384 (+ 1299us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.954420 (+ 36us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.954425 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.954463 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.958076 (+ 3613us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.959611 (+ 1535us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.959619 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.959623 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.959805 (+ 182us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.960523 (+ 718us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.960554 (+ 31us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.960559 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.961033 (+ 474us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.961441 (+ 408us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.962357 (+ 916us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.962777 (+ 420us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":37}
I20251027 14:57:44.972182 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.932271 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38918 (local address 127.0.104.60:43317)
1027 14:57:44.932517 (+ 246us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.932523 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.945598 (+ 13075us) server_negotiation.cc:408] Connection header received
1027 14:57:44.945996 (+ 398us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.946003 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.946090 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.946161 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.953204 (+ 7043us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.954564 (+ 1360us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.956119 (+ 1555us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.956665 (+ 546us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.962355 (+ 5690us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.962402 (+ 47us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.962408 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.962449 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.966809 (+ 4360us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.969904 (+ 3095us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.969912 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.969916 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.970049 (+ 133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.970452 (+ 403us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.970458 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.970462 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.970789 (+ 327us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.971026 (+ 237us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.971552 (+ 526us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.971812 (+ 260us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":127}
I20251027 14:57:44.978734 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.942101 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33580 (local address 127.0.104.61:40605)
1027 14:57:44.942288 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.942294 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.949205 (+ 6911us) server_negotiation.cc:408] Connection header received
1027 14:57:44.949743 (+ 538us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.949750 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.949866 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.949941 (+ 75us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.956369 (+ 6428us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.957588 (+ 1219us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.961065 (+ 3477us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.961580 (+ 515us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.969244 (+ 7664us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.969288 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.969295 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.969346 (+ 51us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.973967 (+ 4621us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.975120 (+ 1153us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.975127 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.975131 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.975275 (+ 144us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.975876 (+ 601us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.975882 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.975886 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.976424 (+ 538us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.976857 (+ 433us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.977701 (+ 844us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.978142 (+ 441us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":62}
Oct 27 14:57:44 dist-test-slave-hxxt krb5kdc[23621](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577064, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
I20251027 14:57:44.981236 23832 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [term 1 FOLLOWER]: Refusing update from remote peer d5ac7e7dff3b4d6da6c515ffe19aa29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:57:44.982378 23885 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:57:44.989588 23784 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.960304 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:43317 (local address 127.0.0.1:38924)
1027 14:57:44.960753 (+ 449us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:44.960773 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:44.960879 (+ 106us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:44.972752 (+ 11873us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:44.972759 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:44.973226 (+ 467us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:44.973663 (+ 437us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.973679 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.975633 (+ 1954us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.975640 (+ 7us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:44.976809 (+ 1169us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:44.976825 (+ 16us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.976965 (+ 140us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.977693 (+ 728us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:44.977719 (+ 26us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:44.981165 (+ 3446us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:44.986750 (+ 5585us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.986756 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.986761 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.987255 (+ 494us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.987637 (+ 382us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:44.987643 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:44.987646 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:44.987744 (+ 98us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:44.988476 (+ 732us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:44.988481 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:44.988626 (+ 145us) client_negotiation.cc:769] Sending connection context
1027 14:57:44.988898 (+ 272us) client_negotiation.cc:240] Negotiation successful
1027 14:57:44.989208 (+ 310us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":335,"spinlock_wait_cycles":44928}
I20251027 14:57:44.990732 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:44.961079 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38924 (local address 127.0.104.60:43317)
1027 14:57:44.972424 (+ 11345us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:44.972429 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:44.972446 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:57:44.972491 (+ 45us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:44.972495 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:44.972563 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:44.974112 (+ 1549us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:44.974264 (+ 152us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.975478 (+ 1214us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:44.978834 (+ 3356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:44.981835 (+ 3001us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:44.982038 (+ 203us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:44.982068 (+ 30us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:44.982073 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:44.982101 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:44.986576 (+ 4475us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.987406 (+ 830us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.987412 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.987416 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.987497 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:44.987875 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:44.987880 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:44.987884 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:44.988218 (+ 334us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:44.988415 (+ 197us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:44.990008 (+ 1593us) server_negotiation.cc:300] Negotiation successful
1027 14:57:44.990254 (+ 246us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":11235,"thread_start_us":11724,"threads_started":1}
I20251027 14:57:44.997255 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: Bootstrap starting.
I20251027 14:57:45.001578 23780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:45.002993 23780 log.cc:826] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: Log is configured to *not* fsync() on all Append() calls
W20251027 14:57:45.010424 23811 tablet.cc:2378] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:57:45.015010 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ac7e7dff3b4d6da6c515ffe19aa29e. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.015291 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:45.017998 23901 mvcc.cc:204] Tried to move back new op lower bound from 7215419658134372352 to 7215419657818587136. Current Snapshot: MvccSnapshot[applied={T|T < 7215419658134372352}]
I20251027 14:57:45.019021 23904 mvcc.cc:204] Tried to move back new op lower bound from 7215419658134372352 to 7215419657818587136. Current Snapshot: MvccSnapshot[applied={T|T < 7215419658134372352}]
I20251027 14:57:45.019135 23780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: No bootstrap required, opened a new log
I20251027 14:57:45.030779 23780 raft_consensus.cc:359] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:45.031250 23865 sys_catalog.cc:455] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.031458 23865 sys_catalog.cc:458] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:45.031435 23780 raft_consensus.cc:740] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 49bf73c91c3e4e53a9d428cdae6b5ed2, State: Initialized, Role: FOLLOWER
I20251027 14:57:45.032336 23780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } }
I20251027 14:57:45.037242 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.037467 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:45.041457 23891 catalog_manager.cc:1357] Generated new cluster ID: 4a1a9dcab8894b8f98d868818c0b915f
I20251027 14:57:45.041556 23891 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:45.044932 23909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.045209 23909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:45.046216 23780 sys_catalog.cc:565] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:45.046257 23755 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2"
I20251027 14:57:45.046675 23755 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:45.050226 23755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d5ac7e7dff3b4d6da6c515ffe19aa29e in term 1.
I20251027 14:57:45.051538 23754 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [term 1 FOLLOWER]: Refusing update from remote peer d5ac7e7dff3b4d6da6c515ffe19aa29e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:57:45.052352 23879 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [LEADER]: Connected to new peer: Peer: permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:57:45.080166 23917 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.081034 23917 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:45.080971 23909 sys_catalog.cc:455] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d5ac7e7dff3b4d6da6c515ffe19aa29e. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.081296 23909 sys_catalog.cc:458] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:45.082230 23885 sys_catalog.cc:455] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d5ac7e7dff3b4d6da6c515ffe19aa29e" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 38221 } } peers { permanent_uuid: "49bf73c91c3e4e53a9d428cdae6b5ed2" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40605 } } peers { permanent_uuid: "29ad36b2fdbc4f058ea6a2209708b12f" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 43317 } } }
I20251027 14:57:45.082464 23885 sys_catalog.cc:458] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:45.090056 23891 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:45.091711 23891 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:45.112013 23891 catalog_manager.cc:6022] T 00000000000000000000000000000000 P d5ac7e7dff3b4d6da6c515ffe19aa29e: Generated new TSK 0
I20251027 14:57:45.113137 23891 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:45.117074 23925 catalog_manager.cc:1269] Loaded cluster ID: 4a1a9dcab8894b8f98d868818c0b915f
I20251027 14:57:45.117233 23925 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: loading cluster ID for follower catalog manager: success
I20251027 14:57:45.126067 23925 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: acquiring CA information for follower catalog manager: success
I20251027 14:57:45.128741 23925 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 49bf73c91c3e4e53a9d428cdae6b5ed2: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251027 14:57:45.249972 23863 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.228129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53842 (local address 127.0.104.62:38221)
1027 14:57:45.228335 (+ 206us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.228342 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.246004 (+ 17662us) server_negotiation.cc:408] Connection header received
1027 14:57:45.247436 (+ 1432us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.247443 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.247538 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.247619 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.249634 (+ 2015us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53842: BlockingRecv error: recv got EOF from 127.0.0.1:53842 (error 108)
Metrics: {"server-negotiator.queue_time_us":77}
W20251027 14:57:45.251014 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.230288 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38936 (local address 127.0.104.60:43317)
1027 14:57:45.230463 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.230469 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.249188 (+ 18719us) server_negotiation.cc:408] Connection header received
1027 14:57:45.249442 (+ 254us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.249449 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.249529 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.249592 (+ 63us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.250734 (+ 1142us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38936: BlockingRecv error: recv got EOF from 127.0.0.1:38936 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
W20251027 14:57:45.252213 23785 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.243713 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33588 (local address 127.0.104.61:40605)
1027 14:57:45.243905 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.243914 (+ 9us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.250680 (+ 6766us) server_negotiation.cc:408] Connection header received
1027 14:57:45.250838 (+ 158us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.250855 (+ 17us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.250933 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.251028 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.251958 (+ 930us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33588: BlockingRecv error: recv got EOF from 127.0.0.1:33588 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:57:45.294431 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.279526 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33604 (local address 127.0.104.61:40605)
1027 14:57:45.279717 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.279724 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.286821 (+ 7097us) server_negotiation.cc:408] Connection header received
1027 14:57:45.287391 (+ 570us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.287399 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.287608 (+ 209us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.287744 (+ 136us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.288756 (+ 1012us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.290196 (+ 1440us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.291894 (+ 1698us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.292468 (+ 574us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.292915 (+ 447us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.293640 (+ 725us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.293762 (+ 122us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.294007 (+ 245us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.294120 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20251027 14:57:45.337421 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.325129 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38946 (local address 127.0.104.60:43317)
1027 14:57:45.325318 (+ 189us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.325325 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.335583 (+ 10258us) server_negotiation.cc:408] Connection header received
1027 14:57:45.335829 (+ 246us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.335836 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.335967 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.336074 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.337156 (+ 1082us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38946: BlockingRecv error: recv got EOF from 127.0.0.1:38946 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:57:45.387887 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.373446 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33614 (local address 127.0.104.61:40605)
1027 14:57:45.373637 (+ 191us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.373644 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.380971 (+ 7327us) server_negotiation.cc:408] Connection header received
1027 14:57:45.381539 (+ 568us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.381547 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.381665 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.381750 (+ 85us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.382821 (+ 1071us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.384257 (+ 1436us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.386040 (+ 1783us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.386676 (+ 636us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.387010 (+ 334us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.387205 (+ 195us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.387267 (+ 62us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.387475 (+ 208us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.387588 (+ 113us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":73}
W20251027 14:57:45.434497 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.421679 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38952 (local address 127.0.104.60:43317)
1027 14:57:45.421844 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.421850 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.432482 (+ 10632us) server_negotiation.cc:408] Connection header received
1027 14:57:45.433001 (+ 519us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.433008 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.433102 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.433182 (+ 80us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.434269 (+ 1087us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38952: BlockingRecv error: recv got EOF from 127.0.0.1:38952 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:57:45.481564 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.470388 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33618 (local address 127.0.104.61:40605)
1027 14:57:45.470571 (+ 183us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.470578 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.476928 (+ 6350us) server_negotiation.cc:408] Connection header received
1027 14:57:45.477136 (+ 208us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.477141 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.477238 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.477327 (+ 89us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.478155 (+ 828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.479114 (+ 959us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.480322 (+ 1208us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.480751 (+ 429us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.480810 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.480923 (+ 113us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.481004 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.481159 (+ 155us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.481251 (+ 92us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":69}
W20251027 14:57:45.492007 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.490354 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38956 (local address 127.0.104.60:43317)
1027 14:57:45.490524 (+ 170us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.490530 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.490620 (+ 90us) server_negotiation.cc:408] Connection header received
1027 14:57:45.490873 (+ 253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.490880 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.490954 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.491033 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.491819 (+ 786us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38956: BlockingRecv error: recv got EOF from 127.0.0.1:38956 (error 108)
Metrics: {"server-negotiator.queue_time_us":67}
I20251027 14:57:45.505990 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.501090 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33624 (local address 127.0.104.61:40605)
1027 14:57:45.501294 (+ 204us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.501300 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.501785 (+ 485us) server_negotiation.cc:408] Connection header received
1027 14:57:45.501962 (+ 177us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.501969 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.502036 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.502112 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.502711 (+ 599us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.503681 (+ 970us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.504818 (+ 1137us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.505256 (+ 438us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.505314 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.505443 (+ 129us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.505522 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.505652 (+ 130us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.505738 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":90}
W20251027 14:57:45.513463 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.511284 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38958 (local address 127.0.104.60:43317)
1027 14:57:45.511542 (+ 258us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.511550 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.512236 (+ 686us) server_negotiation.cc:408] Connection header received
1027 14:57:45.512435 (+ 199us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.512442 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.512516 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.512594 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.513259 (+ 665us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38958: BlockingRecv error: recv got EOF from 127.0.0.1:38958 (error 108)
Metrics: {"server-negotiator.queue_time_us":143}
I20251027 14:57:45.529461 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.524401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33640 (local address 127.0.104.61:40605)
1027 14:57:45.524598 (+ 197us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.524605 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.525099 (+ 494us) server_negotiation.cc:408] Connection header received
1027 14:57:45.525305 (+ 206us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.525310 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.525412 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.525488 (+ 76us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.526081 (+ 593us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.527027 (+ 946us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.528142 (+ 1115us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.528632 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.528698 (+ 66us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.528829 (+ 131us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.528919 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.529058 (+ 139us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.529149 (+ 91us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20251027 14:57:45.536981 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.534999 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38972 (local address 127.0.104.60:43317)
1027 14:57:45.535180 (+ 181us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.535187 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.535680 (+ 493us) server_negotiation.cc:408] Connection header received
1027 14:57:45.535860 (+ 180us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.535865 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.535949 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.536030 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.536741 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38972: BlockingRecv error: recv got EOF from 127.0.0.1:38972 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
I20251027 14:57:45.561554 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.556074 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33646 (local address 127.0.104.61:40605)
1027 14:57:45.556261 (+ 187us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.556269 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.556755 (+ 486us) server_negotiation.cc:408] Connection header received
1027 14:57:45.557059 (+ 304us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.557064 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.557162 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.557241 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.557892 (+ 651us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.558913 (+ 1021us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.560195 (+ 1282us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.560730 (+ 535us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.560796 (+ 66us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.560940 (+ 144us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.561023 (+ 83us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.561163 (+ 140us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.561263 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20251027 14:57:45.569819 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.567476 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38976 (local address 127.0.104.60:43317)
1027 14:57:45.567698 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.567705 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.568235 (+ 530us) server_negotiation.cc:408] Connection header received
1027 14:57:45.568450 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.568544 (+ 94us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.568653 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.568840 (+ 187us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.569570 (+ 730us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38976: BlockingRecv error: recv got EOF from 127.0.0.1:38976 (error 108)
Metrics: {"server-negotiator.queue_time_us":70}
I20251027 14:57:45.611251 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.605145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33654 (local address 127.0.104.61:40605)
1027 14:57:45.605339 (+ 194us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.605346 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.605821 (+ 475us) server_negotiation.cc:408] Connection header received
1027 14:57:45.606038 (+ 217us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.606044 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.606117 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.606197 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.606919 (+ 722us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.608261 (+ 1342us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.609874 (+ 1613us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.610367 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.610447 (+ 80us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.610588 (+ 141us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.610670 (+ 82us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.610819 (+ 149us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.610923 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":68}
W20251027 14:57:45.622581 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.620813 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38992 (local address 127.0.104.60:43317)
1027 14:57:45.621015 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.621021 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.621258 (+ 237us) server_negotiation.cc:408] Connection header received
1027 14:57:45.621408 (+ 150us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.621416 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.621519 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.621647 (+ 128us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.622348 (+ 701us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38992: BlockingRecv error: recv got EOF from 127.0.0.1:38992 (error 108)
Metrics: {"server-negotiator.queue_time_us":86}
I20251027 14:57:45.694506 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.689819 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33668 (local address 127.0.104.61:40605)
1027 14:57:45.690041 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.690048 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.690324 (+ 276us) server_negotiation.cc:408] Connection header received
1027 14:57:45.690522 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.690527 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.690595 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.690674 (+ 79us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.691302 (+ 628us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.692207 (+ 905us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.693385 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.693788 (+ 403us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.693847 (+ 59us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.693962 (+ 115us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.694042 (+ 80us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.694175 (+ 133us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.694270 (+ 95us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":84}
W20251027 14:57:45.701669 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.699736 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38994 (local address 127.0.104.60:43317)
1027 14:57:45.699952 (+ 216us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.699958 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.700179 (+ 221us) server_negotiation.cc:408] Connection header received
1027 14:57:45.700362 (+ 183us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.700369 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.700478 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.700554 (+ 76us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.701435 (+ 881us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38994: BlockingRecv error: recv got EOF from 127.0.0.1:38994 (error 108)
Metrics: {"server-negotiator.queue_time_us":57}
I20251027 14:57:45.838578 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:45.833930 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33674 (local address 127.0.104.61:40605)
1027 14:57:45.834105 (+ 175us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.834110 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.834357 (+ 247us) server_negotiation.cc:408] Connection header received
1027 14:57:45.834514 (+ 157us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.834519 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.834590 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.834681 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:45.835420 (+ 739us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.836344 (+ 924us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:45.837586 (+ 1242us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:45.837940 (+ 354us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:45.837997 (+ 57us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:45.838123 (+ 126us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:45.838196 (+ 73us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:45.838296 (+ 100us) server_negotiation.cc:300] Negotiation successful
1027 14:57:45.838377 (+ 81us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":55}
W20251027 14:57:45.845829 23860 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:45.844106 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39006 (local address 127.0.104.60:43317)
1027 14:57:45.844261 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:45.844266 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:45.844542 (+ 276us) server_negotiation.cc:408] Connection header received
1027 14:57:45.844702 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:45.844707 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:45.844804 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:45.844889 (+ 85us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:45.845563 (+ 674us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39006: BlockingRecv error: recv got EOF from 127.0.0.1:39006 (error 108)
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:57:45.868867 23876 catalog_manager.cc:1269] Loaded cluster ID: 4a1a9dcab8894b8f98d868818c0b915f
I20251027 14:57:45.868963 23876 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: loading cluster ID for follower catalog manager: success
I20251027 14:57:45.871441 23876 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: acquiring CA information for follower catalog manager: success
I20251027 14:57:45.873206 23876 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 29ad36b2fdbc4f058ea6a2209708b12f: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 14:57:46.110162 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:46.105695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33688 (local address 127.0.104.61:40605)
1027 14:57:46.105917 (+ 222us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:46.105924 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:46.106160 (+ 236us) server_negotiation.cc:408] Connection header received
1027 14:57:46.106312 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:46.106316 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:46.106399 (+ 83us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:46.106479 (+ 80us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:46.107051 (+ 572us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.107960 (+ 909us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:46.109138 (+ 1178us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.109449 (+ 311us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:46.109502 (+ 53us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:46.109612 (+ 110us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:46.109679 (+ 67us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:46.109795 (+ 116us) server_negotiation.cc:300] Negotiation successful
1027 14:57:46.109906 (+ 111us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":63}
I20251027 14:57:46.120731 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:46.115703 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39008 (local address 127.0.104.60:43317)
1027 14:57:46.115902 (+ 199us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:46.115908 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:46.116053 (+ 145us) server_negotiation.cc:408] Connection header received
1027 14:57:46.116238 (+ 185us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:46.116244 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:46.116419 (+ 175us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:46.116536 (+ 117us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:46.117136 (+ 600us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.117966 (+ 830us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:46.119100 (+ 1134us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.119470 (+ 370us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:46.119615 (+ 145us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:46.120133 (+ 518us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:46.120266 (+ 133us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:46.120363 (+ 97us) server_negotiation.cc:300] Negotiation successful
1027 14:57:46.120479 (+ 116us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
I20251027 14:57:46.143561 23860 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:46.126435 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39012 (local address 127.0.104.60:43317)
1027 14:57:46.126818 (+ 383us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:46.126823 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:46.137809 (+ 10986us) server_negotiation.cc:408] Connection header received
1027 14:57:46.138021 (+ 212us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:46.138027 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:46.138115 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:46.138180 (+ 65us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:46.139211 (+ 1031us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.140203 (+ 992us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:46.141931 (+ 1728us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.142402 (+ 471us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:46.142747 (+ 345us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:46.142900 (+ 153us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:46.142960 (+ 60us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:46.143183 (+ 223us) server_negotiation.cc:300] Negotiation successful
1027 14:57:46.143280 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":261}
I20251027 14:57:46.143558 23785 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:46.126631 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:33690 (local address 127.0.104.61:40605)
1027 14:57:46.126946 (+ 315us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:46.126953 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:46.137474 (+ 10521us) server_negotiation.cc:408] Connection header received
1027 14:57:46.137739 (+ 265us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:46.137746 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:46.137836 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:46.137927 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:46.138913 (+ 986us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.140103 (+ 1190us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:46.141931 (+ 1828us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.142395 (+ 464us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:46.142748 (+ 353us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:46.142903 (+ 155us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:46.142961 (+ 58us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:46.143183 (+ 222us) server_negotiation.cc:300] Negotiation successful
1027 14:57:46.143280 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":209}
I20251027 14:57:46.143803 24042 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:46.126212 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53844 (local address 127.0.104.62:38221)
1027 14:57:46.136100 (+ 9888us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:46.136108 (+ 8us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:46.136783 (+ 675us) server_negotiation.cc:408] Connection header received
1027 14:57:46.137027 (+ 244us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:46.137034 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:46.137260 (+ 226us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:46.137459 (+ 199us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:46.138551 (+ 1092us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.139763 (+ 1212us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:46.141221 (+ 1458us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:46.142254 (+ 1033us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:46.142417 (+ 163us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:46.142863 (+ 446us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:46.142973 (+ 110us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:46.143405 (+ 432us) server_negotiation.cc:300] Negotiation successful
1027 14:57:46.143509 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":9748,"thread_start_us":9905,"threads_started":1}
I20251027 14:57:46.149104 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23637
I20251027 14:57:46.150022 23699 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:46.259979 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23637
W20251027 14:57:46.277522 23807 connection.cc:537] server connection from 127.0.0.1:38910 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20251027 14:57:46.277912 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23713
I20251027 14:57:46.278999 23775 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:46.385303 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23713
I20251027 14:57:46.403518 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 23790
I20251027 14:57:46.405153 23852 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:46.509373 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 23790
2025-10-27T14:57:46Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/0 (5410 ms)
[ RUN ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1
Loading random data
Initializing database '/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Oct 27 14:57:46 dist-test-slave-hxxt krb5kdc[24049](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Oct 27 14:57:46 dist-test-slave-hxxt krb5kdc[24049](info): set up 2 sockets
Oct 27 14:57:46 dist-test-slave-hxxt krb5kdc[24049](info): commencing operation
krb5kdc: starting...
W20251027 14:57:48.392709 416 mini_kdc.cc:121] Time spent starting KDC: real 1.847s user 0.001s sys 0.026s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/test-user.keytab.
Oct 27 14:57:48 dist-test-slave-hxxt krb5kdc[24049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2025-10-27T14:57:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2025-10-27T14:57:48Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.62@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.62@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.62 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:48.693339 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36773
--webserver_interface=127.0.104.62
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:36773,127.0.104.61:40075,127.0.104.60:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20251027 14:57:49.033140 24065 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251027 14:57:49.033504 24065 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:49.033586 24065 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:49.033728 24065 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:49.041817 24065 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:49.041946 24065 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:49.042025 24065 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:49.042107 24065 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:49.042164 24065 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:49.052613 24065 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:36773,127.0.104.61:40075,127.0.104.60:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.62
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.62:36773
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.104.62
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:49.054220 24065 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:49.055541 24065 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:49.063197 24071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:49.063264 24070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:49.064224 24073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:49.064759 24065 server_base.cc:1047] running on GCE node
I20251027 14:57:49.065443 24065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:49.066830 24065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:49.068032 24065 hybrid_clock.cc:648] HybridClock initialized: now 1761577069067989 us; error 37 us; skew 500 ppm
I20251027 14:57:49.068354 24065 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:49 dist-test-slave-hxxt krb5kdc[24049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:49.071344 24065 init.cc:377] Logged in from keytab as kudu/127.0.104.62@KRBTEST.COM (short username kudu)
I20251027 14:57:49.072811 24065 webserver.cc:492] Webserver started at http://127.0.104.62:45655/ using document root <none> and password file <none>
I20251027 14:57:49.073490 24065 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:49.073594 24065 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:49.073905 24065 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:49.076514 24065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data/instance:
uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97"
format_stamp: "Formatted at 2025-10-27 14:57:49 on dist-test-slave-hxxt"
server_key: "72c7ecb46bf88c33daa4d31281f21100"
server_key_iv: "839511f4bc9a39180bded8ef547de627"
server_key_version: "encryptionkey@0"
I20251027 14:57:49.077301 24065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal/instance:
uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97"
format_stamp: "Formatted at 2025-10-27 14:57:49 on dist-test-slave-hxxt"
server_key: "72c7ecb46bf88c33daa4d31281f21100"
server_key_iv: "839511f4bc9a39180bded8ef547de627"
server_key_version: "encryptionkey@0"
I20251027 14:57:49.082401 24065 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:57:49.085537 24080 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:49.086766 24065 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:57:49.086915 24065 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal
uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97"
format_stamp: "Formatted at 2025-10-27 14:57:49 on dist-test-slave-hxxt"
server_key: "72c7ecb46bf88c33daa4d31281f21100"
server_key_iv: "839511f4bc9a39180bded8ef547de627"
server_key_version: "encryptionkey@0"
I20251027 14:57:49.087090 24065 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:49.104741 24065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:49.108605 24065 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:49.108848 24065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:49.125727 24065 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.62:36773
I20251027 14:57:49.125756 24131 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.62:36773 every 8 connection(s)
I20251027 14:57:49.127161 24065 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/data/info.pb
I20251027 14:57:49.131228 24132 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:49.137344 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 24065
I20251027 14:57:49.137506 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-0/wal/instance
I20251027 14:57:49.137871 416 external_mini_cluster.cc:1468] Setting key 58edc69e41d2a619f08ef938abd83b2a
I20251027 14:57:49.137488 24132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:49 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:49.158154 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.140400 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36773 (local address 127.0.0.1:53336)
1027 14:57:49.141430 (+ 1030us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.141462 (+ 32us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:49.142799 (+ 1337us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:49.143636 (+ 837us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:49.143643 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:49.144367 (+ 724us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:49.144899 (+ 532us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.144918 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.146591 (+ 1673us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.146596 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:49.147319 (+ 723us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.147329 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.147589 (+ 260us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.148352 (+ 763us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:49.148393 (+ 41us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:49.150461 (+ 2068us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:49.154155 (+ 3694us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.154165 (+ 10us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.154177 (+ 12us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.154690 (+ 513us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.155120 (+ 430us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.155124 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.155126 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.155225 (+ 99us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.156153 (+ 928us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:49.156159 (+ 6us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:49.156344 (+ 185us) client_negotiation.cc:769] Sending connection context
1027 14:57:49.156642 (+ 298us) client_negotiation.cc:240] Negotiation successful
1027 14:57:49.157040 (+ 398us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":862,"thread_start_us":683,"threads_started":1}
I20251027 14:57:49.158177 24134 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.140401 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53336 (local address 127.0.104.62:36773)
1027 14:57:49.141179 (+ 778us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:49.141194 (+ 15us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:49.141575 (+ 381us) server_negotiation.cc:408] Connection header received
1027 14:57:49.143176 (+ 1601us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:49.143182 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:49.143417 (+ 235us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:49.143604 (+ 187us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:49.145093 (+ 1489us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.146445 (+ 1352us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.147442 (+ 997us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.147954 (+ 512us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.150605 (+ 2651us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:49.150662 (+ 57us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:49.150674 (+ 12us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:49.150699 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:49.154009 (+ 3310us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.154857 (+ 848us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.154867 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.154871 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.154999 (+ 128us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.155366 (+ 367us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.155372 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.155375 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.156007 (+ 632us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:49.156195 (+ 188us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:49.156777 (+ 582us) server_negotiation.cc:300] Negotiation successful
1027 14:57:49.157237 (+ 460us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":114,"server-negotiator.queue_time_us":583,"thread_start_us":375,"threads_started":1}
I20251027 14:57:49.161195 24132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:49.162304 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.161946 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38428)
1027 14:57:49.162092 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.162185 (+ 93us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20251027 14:57:49.162532 24082 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
WARNING: no policy specified for kudu/127.0.104.61@KRBTEST.COM; defaulting to no policy
I20251027 14:57:49.163997 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.163633 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38434)
1027 14:57:49.163796 (+ 163us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.163867 (+ 71us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251027 14:57:49.164165 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40075: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.61@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.61@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.61@KRBTEST.COM" created.
I20251027 14:57:49.213474 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } attempt: 1
I20251027 14:57:49.214644 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.214160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38444)
1027 14:57:49.214370 (+ 210us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.214488 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20251027 14:57:49.215688 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.215337 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38448)
1027 14:57:49.215459 (+ 122us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.215529 (+ 70us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":38}
W20251027 14:57:49.215864 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40075: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.61 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:49.240650 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40075
--webserver_interface=127.0.104.61
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.61
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:36773,127.0.104.61:40075,127.0.104.60:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:57:49.284070 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } attempt: 2
I20251027 14:57:49.285174 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.284732 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38458)
1027 14:57:49.284941 (+ 209us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.285027 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":64}
I20251027 14:57:49.286082 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.285738 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38472)
1027 14:57:49.285869 (+ 131us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.285941 (+ 72us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251027 14:57:49.286263 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40075: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
I20251027 14:57:49.377480 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } attempt: 3
I20251027 14:57:49.378486 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.378095 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38480)
1027 14:57:49.378273 (+ 178us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.378357 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":58}
I20251027 14:57:49.379391 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.379074 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38494)
1027 14:57:49.379199 (+ 125us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.379263 (+ 64us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":39}
W20251027 14:57:49.379559 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40075: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
I20251027 14:57:49.522789 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } attempt: 4
I20251027 14:57:49.523955 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.523447 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38496)
1027 14:57:49.523734 (+ 287us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.523819 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":154}
I20251027 14:57:49.524878 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.524554 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38500)
1027 14:57:49.524681 (+ 127us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.524758 (+ 77us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":40}
W20251027 14:57:49.525074 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.61:40075: Network error: Client connection negotiation failed: client connection to 127.0.104.61:40075: connect: Connection refused (error 111)
W20251027 14:57:49.561849 24141 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251027 14:57:49.562196 24141 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:49.562275 24141 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:49.562413 24141 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:49.570744 24141 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:49.570864 24141 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:49.570940 24141 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:49.571022 24141 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:49.571079 24141 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:49.581470 24141 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:36773,127.0.104.61:40075,127.0.104.60:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.61
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.61:40075
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.104.61
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:49.583057 24141 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:49.584347 24141 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:49.592084 24146 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:49.592124 24147 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:49.592499 24149 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:49.593097 24141 server_base.cc:1047] running on GCE node
I20251027 14:57:49.593765 24141 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:49.595009 24141 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:49.596199 24141 hybrid_clock.cc:648] HybridClock initialized: now 1761577069596147 us; error 40 us; skew 500 ppm
I20251027 14:57:49.596542 24141 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:49 dist-test-slave-hxxt krb5kdc[24049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:49.599450 24141 init.cc:377] Logged in from keytab as kudu/127.0.104.61@KRBTEST.COM (short username kudu)
I20251027 14:57:49.601014 24141 webserver.cc:492] Webserver started at http://127.0.104.61:45607/ using document root <none> and password file <none>
I20251027 14:57:49.601776 24141 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:49.601917 24141 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:49.602243 24141 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:49.604923 24141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data/instance:
uuid: "51975d0d4311461fb8c1c9d755f5cfe1"
format_stamp: "Formatted at 2025-10-27 14:57:49 on dist-test-slave-hxxt"
server_key: "7b2b6abd66ca647628ee386be2e5694e"
server_key_iv: "c49251d4470215f3414510ae24c0cc88"
server_key_version: "encryptionkey@0"
I20251027 14:57:49.605667 24141 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal/instance:
uuid: "51975d0d4311461fb8c1c9d755f5cfe1"
format_stamp: "Formatted at 2025-10-27 14:57:49 on dist-test-slave-hxxt"
server_key: "7b2b6abd66ca647628ee386be2e5694e"
server_key_iv: "c49251d4470215f3414510ae24c0cc88"
server_key_version: "encryptionkey@0"
I20251027 14:57:49.611121 24141 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20251027 14:57:49.614799 24156 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:49.616313 24141 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20251027 14:57:49.616539 24141 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal
uuid: "51975d0d4311461fb8c1c9d755f5cfe1"
format_stamp: "Formatted at 2025-10-27 14:57:49 on dist-test-slave-hxxt"
server_key: "7b2b6abd66ca647628ee386be2e5694e"
server_key_iv: "c49251d4470215f3414510ae24c0cc88"
server_key_version: "encryptionkey@0"
I20251027 14:57:49.616768 24141 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:49.637537 24141 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:49.641670 24141 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:49.641948 24141 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:49.660825 24141 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.61:40075
I20251027 14:57:49.660843 24207 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.61:40075 every 8 connection(s)
I20251027 14:57:49.662302 24141 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/data/info.pb
I20251027 14:57:49.664644 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 24141
I20251027 14:57:49.664809 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-1/wal/instance
I20251027 14:57:49.665180 416 external_mini_cluster.cc:1468] Setting key 510140974ce04e5c02c41241c8cf4364
I20251027 14:57:49.666465 24208 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:49.672945 24208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:49 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.104.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.104.60@KRBTEST.COM" created.
I20251027 14:57:49.692191 24211 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.675665 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53342 (local address 127.0.104.62:36773)
1027 14:57:49.676139 (+ 474us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:49.676146 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:49.677169 (+ 1023us) server_negotiation.cc:408] Connection header received
1027 14:57:49.678559 (+ 1390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:49.678566 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:49.678646 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:49.678764 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:49.680440 (+ 1676us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.681617 (+ 1177us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.682808 (+ 1191us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.683252 (+ 444us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.686202 (+ 2950us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:49.686246 (+ 44us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:49.686251 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:49.686292 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:49.689184 (+ 2892us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.689944 (+ 760us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.689948 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.689950 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.690024 (+ 74us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.690345 (+ 321us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.690350 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.690353 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.690625 (+ 272us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:49.690741 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:49.691627 (+ 886us) server_negotiation.cc:300] Negotiation successful
1027 14:57:49.691826 (+ 199us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":355,"thread_start_us":227,"threads_started":1}
I20251027 14:57:49.693437 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.676174 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36773 (local address 127.0.0.1:53342)
1027 14:57:49.676992 (+ 818us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.677046 (+ 54us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:49.678300 (+ 1254us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:49.678989 (+ 689us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:49.678997 (+ 8us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:49.679564 (+ 567us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:49.680245 (+ 681us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.680264 (+ 19us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.681776 (+ 1512us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.681781 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:49.682677 (+ 896us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.682689 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.682902 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.683543 (+ 641us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:49.683568 (+ 25us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:49.685975 (+ 2407us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:49.689305 (+ 3330us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.689312 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.689327 (+ 15us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.689831 (+ 504us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.690103 (+ 272us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.690107 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.690121 (+ 14us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.690243 (+ 122us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.690758 (+ 515us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:49.690766 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:49.691215 (+ 449us) client_negotiation.cc:769] Sending connection context
1027 14:57:49.691568 (+ 353us) client_negotiation.cc:240] Negotiation successful
1027 14:57:49.692095 (+ 527us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":546,"thread_start_us":366,"threads_started":1}
I20251027 14:57:49.695330 24208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:49 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:49.708154 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.696077 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38512)
1027 14:57:49.696233 (+ 156us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.696253 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:49.696369 (+ 116us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:49.697211 (+ 842us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:49.697215 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:49.697533 (+ 318us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:49.697851 (+ 318us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.697862 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.699794 (+ 1932us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.699799 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:49.700552 (+ 753us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.700563 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.700716 (+ 153us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.701314 (+ 598us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:49.701331 (+ 17us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:49.703256 (+ 1925us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:49.706113 (+ 2857us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.706118 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.706123 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.706414 (+ 291us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.706678 (+ 264us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.706689 (+ 11us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.706692 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.706762 (+ 70us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.707262 (+ 500us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:49.707267 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:49.707396 (+ 129us) client_negotiation.cc:769] Sending connection context
1027 14:57:49.707525 (+ 129us) client_negotiation.cc:240] Negotiation successful
1027 14:57:49.707816 (+ 291us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":53}
I20251027 14:57:49.708287 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.696145 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38512 (local address 127.0.104.61:40075)
1027 14:57:49.696704 (+ 559us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:49.696711 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:49.696739 (+ 28us) server_negotiation.cc:408] Connection header received
1027 14:57:49.696787 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:49.696794 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:49.697024 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:49.697189 (+ 165us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:49.698028 (+ 839us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.699652 (+ 1624us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.700724 (+ 1072us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.701115 (+ 391us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.703390 (+ 2275us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:49.703445 (+ 55us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:49.703459 (+ 14us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:49.703484 (+ 25us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:49.706001 (+ 2517us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.706524 (+ 523us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.706530 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.706532 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.706592 (+ 60us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.706844 (+ 252us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.706849 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.706852 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.707153 (+ 301us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:49.707316 (+ 163us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:49.707651 (+ 335us) server_negotiation.cc:300] Negotiation successful
1027 14:57:49.707943 (+ 292us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"mutex_wait_us":23,"server-negotiator.queue_time_us":426,"thread_start_us":286,"threads_started":1}
I20251027 14:57:49.710914 24208 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:49.712077 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.711600 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:58996)
1027 14:57:49.711780 (+ 180us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.711913 (+ 133us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":61}
W20251027 14:57:49.712368 24159 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:49.713936 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.713553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59006)
1027 14:57:49.713704 (+ 151us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.713783 (+ 79us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":49}
W20251027 14:57:49.714143 24208 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.104.60@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.104.60@KRBTEST.COM" created.
I20251027 14:57:49.763486 24208 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 1
I20251027 14:57:49.764602 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.764132 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59016)
1027 14:57:49.764345 (+ 213us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.764451 (+ 106us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":73}
I20251027 14:57:49.765548 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.765248 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59024)
1027 14:57:49.765363 (+ 115us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.765426 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251027 14:57:49.765736 24208 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.104.60 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab.
I20251027 14:57:49.769312 416 external_mini_cluster.cc:1366] Running /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
/tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42789
--webserver_interface=127.0.104.60
--webserver_port=0
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.60
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.104.20:41769
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.104.62:36773,127.0.104.61:40075,127.0.104.60:42789
--encrypt_data_at_rest=true
--rpc_trace_negotiation
--enable_leader_failure_detection=false
--rpc_authentication=required
--rpc_encryption=required with env {KRB5CCNAME=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20251027 14:57:49.824307 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } attempt: 5
Oct 27 14:57:49 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:49.833932 24208 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 2
I20251027 14:57:49.834836 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.834401 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59036)
1027 14:57:49.834569 (+ 168us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.834653 (+ 84us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":57}
I20251027 14:57:49.835855 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.835512 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59050)
1027 14:57:49.835658 (+ 146us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.835744 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":52}
W20251027 14:57:49.836129 24208 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:49.837471 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.825094 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38528 (local address 127.0.104.61:40075)
1027 14:57:49.825260 (+ 166us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:49.825266 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:49.825342 (+ 76us) server_negotiation.cc:408] Connection header received
1027 14:57:49.825532 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:49.825537 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:49.825602 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:49.825711 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:49.826707 (+ 996us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.827522 (+ 815us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.828600 (+ 1078us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.829087 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.831660 (+ 2573us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:49.831708 (+ 48us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:49.831714 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:49.831755 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:49.835070 (+ 3315us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.835712 (+ 642us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.835719 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.835722 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.835830 (+ 108us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:49.836172 (+ 342us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:49.836178 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:49.836182 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:49.836477 (+ 295us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:49.836627 (+ 150us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:49.836974 (+ 347us) server_negotiation.cc:300] Negotiation successful
1027 14:57:49.837197 (+ 223us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":71}
I20251027 14:57:49.837491 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.825043 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38528)
1027 14:57:49.825241 (+ 198us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.825261 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:49.825409 (+ 148us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:49.825732 (+ 323us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:49.825737 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:49.826150 (+ 413us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:49.826544 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.826556 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.827654 (+ 1098us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:49.827660 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:49.828462 (+ 802us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:49.828474 (+ 12us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:49.828611 (+ 137us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:49.829351 (+ 740us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:49.829370 (+ 19us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:49.831487 (+ 2117us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:49.835223 (+ 3736us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.835229 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.835232 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.835608 (+ 376us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.835960 (+ 352us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:49.835965 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:49.835969 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:49.836070 (+ 101us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:49.836584 (+ 514us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:49.836589 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:49.836747 (+ 158us) client_negotiation.cc:769] Sending connection context
1027 14:57:49.836862 (+ 115us) client_negotiation.cc:240] Negotiation successful
1027 14:57:49.837157 (+ 295us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":65}
I20251027 14:57:49.838405 24132 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:49.839329 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.839009 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59054)
1027 14:57:49.839145 (+ 136us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.839206 (+ 61us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":47}
I20251027 14:57:49.840209 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.839947 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59056)
1027 14:57:49.840048 (+ 101us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.840115 (+ 67us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":36}
W20251027 14:57:49.840415 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:49.891588 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 1
I20251027 14:57:49.892750 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.892160 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59064)
1027 14:57:49.892419 (+ 259us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.892537 (+ 118us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":74}
I20251027 14:57:49.893810 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.893364 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59080)
1027 14:57:49.893535 (+ 171us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.893621 (+ 86us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251027 14:57:49.894039 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:49.927367 24208 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 3
I20251027 14:57:49.928500 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.928005 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59084)
1027 14:57:49.928224 (+ 219us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.928304 (+ 80us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20251027 14:57:49.929508 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.929110 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59092)
1027 14:57:49.929315 (+ 205us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.929405 (+ 90us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":112}
W20251027 14:57:49.929754 24208 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:49.962253 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 2
I20251027 14:57:49.963429 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.962861 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59094)
1027 14:57:49.963100 (+ 239us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.963215 (+ 115us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":78}
I20251027 14:57:49.964453 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:49.964018 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59102)
1027 14:57:49.964175 (+ 157us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:49.964275 (+ 100us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":51}
W20251027 14:57:49.964676 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:50.070887 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 3
I20251027 14:57:50.072070 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.071553 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59108)
1027 14:57:50.071763 (+ 210us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.071865 (+ 102us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":69}
I20251027 14:57:50.072944 24208 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 4
I20251027 14:57:50.073093 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.072685 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59112)
1027 14:57:50.072871 (+ 186us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.072956 (+ 85us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":90}
W20251027 14:57:50.073349 24132 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
I20251027 14:57:50.073805 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.073426 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59118)
1027 14:57:50.073618 (+ 192us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.073681 (+ 63us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":115}
I20251027 14:57:50.074693 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.074416 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59122)
1027 14:57:50.074537 (+ 121us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.074594 (+ 57us) negotiation.cc:327] Negotiation complete: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
Metrics: {"client-negotiator.queue_time_us":46}
W20251027 14:57:50.074882 24208 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.104.60:42789: Network error: Client connection negotiation failed: client connection to 127.0.104.60:42789: connect: Connection refused (error 111)
W20251027 14:57:50.095021 24218 flags.cc:425] Enabled unsafe flag: --enable_leader_failure_detection=false
W20251027 14:57:50.095371 24218 flags.cc:425] Enabled unsafe flag: --openssl_security_level_override=0
W20251027 14:57:50.095443 24218 flags.cc:425] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20251027 14:57:50.095566 24218 flags.cc:425] Enabled unsafe flag: --never_fsync=true
W20251027 14:57:50.103559 24218 flags.cc:425] Enabled experimental flag: --ipki_ca_key_size=768
W20251027 14:57:50.103672 24218 flags.cc:425] Enabled experimental flag: --rpc_trace_negotiation=true
W20251027 14:57:50.103747 24218 flags.cc:425] Enabled experimental flag: --ipki_server_key_size=768
W20251027 14:57:50.103824 24218 flags.cc:425] Enabled experimental flag: --tsk_num_rsa_bits=512
W20251027 14:57:50.103883 24218 flags.cc:425] Enabled experimental flag: --rpc_reuseport=true
I20251027 14:57:50.114554 24218 master_runner.cc:386] Master server non-default flags:
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.104.20:41769
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--enable_leader_failure_detection=false
--fs_data_dirs=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.104.62:36773,127.0.104.61:40075,127.0.104.60:42789
--rpc_trace_negotiation=true
--keytab_file=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.104.60
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.104.60:42789
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--rpc_encryption=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.104.60
--webserver_port=0
--webserver_require_spnego=true
--encrypt_data_at_rest=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--log_dir=/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
Master server version:
kudu 1.19.0-SNAPSHOT
revision 1d0cb013a5aad754f2ed06c25f5ec3d8a1873622
build type FASTDEBUG
built by None at 27 Oct 2025 14:43:24 UTC on 24a791456cd2
build id 8744
ASAN enabled
I20251027 14:57:50.116173 24218 env_posix.cc:2264] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20251027 14:57:50.117584 24218 file_cache.cc:492] Constructed file cache file cache with capacity 419430
W20251027 14:57:50.125732 24223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:50.125950 24224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20251027 14:57:50.125970 24226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20251027 14:57:50.127003 24218 server_base.cc:1047] running on GCE node
I20251027 14:57:50.127707 24218 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20251027 14:57:50.129187 24218 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20251027 14:57:50.130402 24218 hybrid_clock.cc:648] HybridClock initialized: now 1761577070130356 us; error 43 us; skew 500 ppm
I20251027 14:57:50.130736 24218 server_base.cc:847] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20251027 14:57:50.133839 24218 init.cc:377] Logged in from keytab as kudu/127.0.104.60@KRBTEST.COM (short username kudu)
I20251027 14:57:50.135385 24218 webserver.cc:492] Webserver started at http://127.0.104.60:46869/ using document root <none> and password file <none>
I20251027 14:57:50.136066 24218 fs_manager.cc:362] Metadata directory not provided
I20251027 14:57:50.136154 24218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20251027 14:57:50.136480 24218 server_base.cc:895] This appears to be a new deployment of Kudu; creating new FS layout
I20251027 14:57:50.139050 24218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data/instance:
uuid: "241c5a1e86c64b94ba365f2f11a8482d"
format_stamp: "Formatted at 2025-10-27 14:57:50 on dist-test-slave-hxxt"
server_key: "56ad29ce95e79f8be2e4be40934391bc"
server_key_iv: "9d4b9065cca8a9bef6b752933b907208"
server_key_version: "encryptionkey@0"
I20251027 14:57:50.139776 24218 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal/instance:
uuid: "241c5a1e86c64b94ba365f2f11a8482d"
format_stamp: "Formatted at 2025-10-27 14:57:50 on dist-test-slave-hxxt"
server_key: "56ad29ce95e79f8be2e4be40934391bc"
server_key_iv: "9d4b9065cca8a9bef6b752933b907208"
server_key_version: "encryptionkey@0"
I20251027 14:57:50.145104 24218 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20251027 14:57:50.148427 24233 log_block_manager.cc:3788] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20251027 14:57:50.149770 24218 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20251027 14:57:50.149922 24218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data,/tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal
uuid: "241c5a1e86c64b94ba365f2f11a8482d"
format_stamp: "Formatted at 2025-10-27 14:57:50 on dist-test-slave-hxxt"
server_key: "56ad29ce95e79f8be2e4be40934391bc"
server_key_iv: "9d4b9065cca8a9bef6b752933b907208"
server_key_version: "encryptionkey@0"
I20251027 14:57:50.150089 24218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20251027 14:57:50.181844 24218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20251027 14:57:50.185941 24218 env_posix.cc:2264] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20251027 14:57:50.186174 24218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20251027 14:57:50.204993 24218 rpc_server.cc:307] RPC server started. Bound to: 127.0.104.60:42789
I20251027 14:57:50.205021 24284 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.104.60:42789 every 8 connection(s)
I20251027 14:57:50.206532 24218 server_base.cc:1179] Dumped server information to /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/data/info.pb
I20251027 14:57:50.210820 24285 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20251027 14:57:50.213429 416 external_mini_cluster.cc:1428] Started /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu as pid 24218
I20251027 14:57:50.213587 416 external_mini_cluster.cc:1442] Reading /tmp/dist-test-taskuw8ijJ/test-tmp/security-itest.0.ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys_1.1761576633729408-416-0/minicluster-data/master-2/wal/instance
I20251027 14:57:50.213968 416 external_mini_cluster.cc:1468] Setting key 7c8703e4bfcdb5a1c8ce946ab969bb96
I20251027 14:57:50.218952 24285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:50.235805 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.221601 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53344 (local address 127.0.104.62:36773)
1027 14:57:50.222896 (+ 1295us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.222901 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.222916 (+ 15us) server_negotiation.cc:408] Connection header received
1027 14:57:50.222973 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.222977 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.223036 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.223134 (+ 98us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.224482 (+ 1348us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.225460 (+ 978us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.226479 (+ 1019us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.226899 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.230064 (+ 3165us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.230130 (+ 66us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.230135 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.230170 (+ 35us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.233221 (+ 3051us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.233802 (+ 581us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.233807 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.233810 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.233896 (+ 86us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.234274 (+ 378us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.234278 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.234280 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.234574 (+ 294us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.234744 (+ 170us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.235211 (+ 467us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.235440 (+ 229us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":1191,"thread_start_us":1096,"threads_started":1}
I20251027 14:57:50.236506 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.215899 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59130 (local address 127.0.104.60:42789)
1027 14:57:50.216763 (+ 864us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.216780 (+ 17us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.216834 (+ 54us) server_negotiation.cc:408] Connection header received
1027 14:57:50.218481 (+ 1647us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.218513 (+ 32us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.219105 (+ 592us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.219654 (+ 549us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.221216 (+ 1562us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.223207 (+ 1991us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.224453 (+ 1246us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.224946 (+ 493us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.228568 (+ 3622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.228644 (+ 76us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.228663 (+ 19us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.228700 (+ 37us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.232087 (+ 3387us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.232844 (+ 757us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.232854 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.232865 (+ 11us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.232989 (+ 124us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.233278 (+ 289us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.233283 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.233286 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.233834 (+ 548us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.234101 (+ 267us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.234626 (+ 525us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.235282 (+ 656us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":624,"thread_start_us":406,"threads_started":1}
I20251027 14:57:50.236526 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.221562 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.62:36773 (local address 127.0.0.1:53344)
1027 14:57:50.222265 (+ 703us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.222312 (+ 47us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:50.222456 (+ 144us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:50.223232 (+ 776us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:50.223239 (+ 7us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:50.223889 (+ 650us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:50.224283 (+ 394us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.224293 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.225598 (+ 1305us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.225603 (+ 5us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:50.226340 (+ 737us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.226351 (+ 11us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.226564 (+ 213us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.227224 (+ 660us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:50.227251 (+ 27us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:50.229878 (+ 2627us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:50.233345 (+ 3467us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.233354 (+ 9us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.233360 (+ 6us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.233690 (+ 330us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.234007 (+ 317us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.234011 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.234014 (+ 3us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.234162 (+ 148us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.234735 (+ 573us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:50.234743 (+ 8us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:50.234895 (+ 152us) client_negotiation.cc:769] Sending connection context
1027 14:57:50.235087 (+ 192us) client_negotiation.cc:240] Negotiation successful
1027 14:57:50.235400 (+ 313us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":475,"mutex_wait_us":59,"thread_start_us":298,"threads_started":1}
I20251027 14:57:50.238404 24285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } has no permanent_uuid. Determining permanent_uuid...
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.62@KRBTEST.COM
I20251027 14:57:50.250483 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.239163 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38536 (local address 127.0.104.61:40075)
1027 14:57:50.239312 (+ 149us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.239317 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.239483 (+ 166us) server_negotiation.cc:408] Connection header received
1027 14:57:50.239654 (+ 171us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.239661 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.239747 (+ 86us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.239826 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.240758 (+ 932us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.241539 (+ 781us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.242472 (+ 933us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.242892 (+ 420us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.245514 (+ 2622us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.245553 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.245557 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.245588 (+ 31us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.248196 (+ 2608us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.248871 (+ 675us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.248877 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.248881 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.248997 (+ 116us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.249292 (+ 295us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.249295 (+ 3us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.249297 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.249516 (+ 219us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.249634 (+ 118us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.250043 (+ 409us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.250229 (+ 186us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:57:50.250558 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.239193 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.61:40075 (local address 127.0.0.1:38536)
1027 14:57:50.239391 (+ 198us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.239406 (+ 15us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:50.239507 (+ 101us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:50.239972 (+ 465us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:50.239975 (+ 3us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:50.240258 (+ 283us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:50.240621 (+ 363us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.240630 (+ 9us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.241672 (+ 1042us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.241676 (+ 4us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:50.242350 (+ 674us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.242360 (+ 10us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.242481 (+ 121us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.243139 (+ 658us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:50.243155 (+ 16us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:50.245375 (+ 2220us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:50.248316 (+ 2941us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.248322 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.248326 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.248773 (+ 447us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.249097 (+ 324us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.249102 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.249104 (+ 2us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.249194 (+ 90us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.249630 (+ 436us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:50.249635 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:50.249818 (+ 183us) client_negotiation.cc:769] Sending connection context
1027 14:57:50.249971 (+ 153us) client_negotiation.cc:240] Negotiation successful
1027 14:57:50.250224 (+ 253us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":111}
I20251027 14:57:50.250566 24132 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 4
I20251027 14:57:50.251765 24285 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } has no permanent_uuid. Determining permanent_uuid...
I20251027 14:57:50.253075 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.239755 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53356 (local address 127.0.104.62:36773)
1027 14:57:50.239922 (+ 167us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.239928 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.239945 (+ 17us) server_negotiation.cc:408] Connection header received
1027 14:57:50.239993 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.239998 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.240062 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.240140 (+ 78us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.241441 (+ 1301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.242209 (+ 768us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.243360 (+ 1151us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.243797 (+ 437us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.247516 (+ 3719us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.247541 (+ 25us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.247544 (+ 3us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.247571 (+ 27us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.250504 (+ 2933us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.251249 (+ 745us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.251254 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.251257 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.251320 (+ 63us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.251658 (+ 338us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.251663 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.251666 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.251956 (+ 290us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.252224 (+ 268us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.252517 (+ 293us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.252760 (+ 243us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":54}
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.62@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577070, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.60@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577068, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.104.61@KRBTEST.COM
I20251027 14:57:50.265905 24135 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.251091 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59144)
1027 14:57:50.251308 (+ 217us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.251326 (+ 18us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:50.251428 (+ 102us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:50.251854 (+ 426us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:50.251859 (+ 5us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:50.252320 (+ 461us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:50.252803 (+ 483us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.252821 (+ 18us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.254536 (+ 1715us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.254542 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:50.255525 (+ 983us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.255538 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.255657 (+ 119us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.256230 (+ 573us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:50.256248 (+ 18us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:50.258799 (+ 2551us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:50.263348 (+ 4549us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.263355 (+ 7us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.263359 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.263796 (+ 437us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.264181 (+ 385us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.264186 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.264190 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.264295 (+ 105us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.264945 (+ 650us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:50.264950 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:50.265096 (+ 146us) client_negotiation.cc:769] Sending connection context
1027 14:57:50.265252 (+ 156us) client_negotiation.cc:240] Negotiation successful
1027 14:57:50.265538 (+ 286us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":112}
I20251027 14:57:50.265933 24288 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.251140 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59144 (local address 127.0.104.60:42789)
1027 14:57:50.251575 (+ 435us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.251579 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.251589 (+ 10us) server_negotiation.cc:408] Connection header received
1027 14:57:50.251629 (+ 40us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.251634 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.251710 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.251809 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.253128 (+ 1319us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.254429 (+ 1301us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.255911 (+ 1482us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.257776 (+ 1865us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.258987 (+ 1211us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.259029 (+ 42us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.259035 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.259067 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.263188 (+ 4121us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.263957 (+ 769us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.263962 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.263965 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.264057 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.264469 (+ 412us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.264474 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.264477 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.264838 (+ 361us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.264969 (+ 131us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.265373 (+ 404us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.265599 (+ 226us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":354}
I20251027 14:57:50.266762 24289 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.252470 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59152)
1027 14:57:50.252996 (+ 526us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.253013 (+ 17us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:50.253102 (+ 89us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:50.253962 (+ 860us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:50.253966 (+ 4us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:50.254430 (+ 464us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:50.254874 (+ 444us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.254887 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.256505 (+ 1618us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.256511 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:50.257440 (+ 929us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.257454 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.257586 (+ 132us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.258220 (+ 634us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:50.258242 (+ 22us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:50.260743 (+ 2501us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:50.264626 (+ 3883us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.264632 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.264636 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.265028 (+ 392us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.265320 (+ 292us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.265324 (+ 4us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.265328 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.265402 (+ 74us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.265904 (+ 502us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:50.265909 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:50.266051 (+ 142us) client_negotiation.cc:769] Sending connection context
1027 14:57:50.266186 (+ 135us) client_negotiation.cc:240] Negotiation successful
1027 14:57:50.266428 (+ 242us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":434}
I20251027 14:57:50.267659 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.252734 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59152 (local address 127.0.104.60:42789)
1027 14:57:50.253714 (+ 980us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.253718 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.253737 (+ 19us) server_negotiation.cc:408] Connection header received
1027 14:57:50.253772 (+ 35us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.253775 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.253829 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.253921 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.255057 (+ 1136us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.256360 (+ 1303us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.258066 (+ 1706us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.259928 (+ 1862us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.260920 (+ 992us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.260986 (+ 66us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.260992 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.261024 (+ 32us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.264496 (+ 3472us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.265132 (+ 636us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.265138 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.265141 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.265220 (+ 79us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.265507 (+ 287us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.265511 (+ 4us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.265513 (+ 2us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.265788 (+ 275us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.266981 (+ 1193us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.267106 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.267338 (+ 232us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":873,"thread_start_us":707,"threads_started":1}
I20251027 14:57:50.269379 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.254971 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38550 (local address 127.0.104.61:40075)
1027 14:57:50.255111 (+ 140us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.255116 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.255182 (+ 66us) server_negotiation.cc:408] Connection header received
1027 14:57:50.255342 (+ 160us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.255348 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.255417 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.255512 (+ 95us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.257046 (+ 1534us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.258076 (+ 1030us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.259145 (+ 1069us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.259543 (+ 398us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.263249 (+ 3706us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.263275 (+ 26us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.263279 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.263307 (+ 28us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.266330 (+ 3023us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.267020 (+ 690us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.267026 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.267030 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.267111 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.267461 (+ 350us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.267466 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.267470 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.267776 (+ 306us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.268675 (+ 899us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.268808 (+ 133us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.269058 (+ 250us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:57:50.271968 24132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Bootstrap starting.
I20251027 14:57:50.272848 24285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: Bootstrap starting.
I20251027 14:57:50.274895 416 external_mini_cluster.cc:949] 0 TS(s) registered with all masters
I20251027 14:57:50.275336 24132 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:50.276033 24285 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:50.276626 24132 log.cc:826] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:50.277025 24285 log.cc:826] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:50.277925 24107 tablet_service.cc:2038] Received Run Leader Election RPC: tablet_id: "00000000000000000000000000000000"
dest_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97"
from {username='test-admin', principal='test-admin@KRBTEST.COM'} at 127.0.0.1:53356
I20251027 14:57:50.279945 24132 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97: No bootstrap required, opened a new log
I20251027 14:57:50.280166 24285 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: No bootstrap required, opened a new log
I20251027 14:57:50.285280 24132 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.285622 24132 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2ebbf2e21a4a4fb5a8ac62f7889b6e97, State: Initialized, Role: FOLLOWER
I20251027 14:57:50.286204 24132 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.287039 24285 raft_consensus.cc:359] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.287386 24285 raft_consensus.cc:740] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 241c5a1e86c64b94ba365f2f11a8482d, State: Initialized, Role: FOLLOWER
I20251027 14:57:50.287302 24294 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.287598 24294 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.287882 24285 consensus_queue.cc:260] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.288589 24132 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:50.288671 24107 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 0 FOLLOWER]: Starting forced leader election (received explicit request)
I20251027 14:57:50.288854 24107 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:50.288872 24295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.289201 24295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.289963 24285 sys_catalog.cc:565] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:50.291687 24107 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 1 FOLLOWER]: Starting forced leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.294188 24107 leader_election.cc:290] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [CANDIDATE]: Term 1 election: Requested vote from peers 51975d0d4311461fb8c1c9d755f5cfe1 (127.0.104.61:40075), 241c5a1e86c64b94ba365f2f11a8482d (127.0.104.60:42789)
I20251027 14:57:50.294476 24260 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "241c5a1e86c64b94ba365f2f11a8482d"
I20251027 14:57:50.294914 24260 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:50.298650 24260 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebbf2e21a4a4fb5a8ac62f7889b6e97 in term 1.
I20251027 14:57:50.299340 24083 leader_election.cc:304] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 241c5a1e86c64b94ba365f2f11a8482d, 2ebbf2e21a4a4fb5a8ac62f7889b6e97; no voters:
I20251027 14:57:50.301666 24294 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 1 FOLLOWER]: Leader election won for term 1
I20251027 14:57:50.302486 24294 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [term 1 LEADER]: Becoming Leader. State: Replica: 2ebbf2e21a4a4fb5a8ac62f7889b6e97, State: Running, Role: LEADER
I20251027 14:57:50.303052 24294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.311861 24306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebbf2e21a4a4fb5a8ac62f7889b6e97. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.312232 24306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:50.323484 24311 catalog_manager.cc:1485] Loading table and tablet metadata into memory...
I20251027 14:57:50.334870 24320 catalog_manager.cc:805] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20251027 14:57:50.340852 24311 catalog_manager.cc:1494] Initializing Kudu cluster ID...
I20251027 14:57:50.337985 24218 master_runner.cc:186] Error getting master registration for 127.0.104.62:36773: OK, instance_id { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" instance_seqno: 1761577069121118 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 14:57:50.344048 24218 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
W20251027 14:57:50.361539 24086 tablet.cc:2378] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20251027 14:57:50.362450 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.338780 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38562 (local address 127.0.104.61:40075)
1027 14:57:50.338957 (+ 177us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.338964 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.345418 (+ 6454us) server_negotiation.cc:408] Connection header received
1027 14:57:50.345687 (+ 269us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.345694 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.345782 (+ 88us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.345855 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.348190 (+ 2335us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.349440 (+ 1250us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.351101 (+ 1661us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.351612 (+ 511us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.354746 (+ 3134us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.354792 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.354798 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.354839 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.358315 (+ 3476us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.359117 (+ 802us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.359124 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.359128 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.359243 (+ 115us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.359579 (+ 336us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.359586 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.359590 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.359923 (+ 333us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.360173 (+ 250us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.360772 (+ 599us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.361033 (+ 261us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":60}
I20251027 14:57:50.365535 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.344873 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59168 (local address 127.0.104.60:42789)
1027 14:57:50.345028 (+ 155us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.345034 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.346012 (+ 978us) server_negotiation.cc:408] Connection header received
1027 14:57:50.346329 (+ 317us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.346336 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.346409 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.346477 (+ 68us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.350152 (+ 3675us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.351305 (+ 1153us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.353862 (+ 2557us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.354352 (+ 490us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.358380 (+ 4028us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.358423 (+ 43us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.358428 (+ 5us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.358464 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.361917 (+ 3453us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.363010 (+ 1093us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.363016 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.363019 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.363111 (+ 92us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.363657 (+ 546us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.363662 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.363666 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.363942 (+ 276us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.364154 (+ 212us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.365012 (+ 858us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.365239 (+ 227us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":50}
I20251027 14:57:50.368595 24260 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [term 1 FOLLOWER]: Refusing update from remote peer 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:57:50.368770 24290 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.338706 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53364 (local address 127.0.104.62:36773)
1027 14:57:50.338863 (+ 157us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.338869 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.344664 (+ 5795us) server_negotiation.cc:408] Connection header received
1027 14:57:50.348220 (+ 3556us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.348227 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.348309 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.348408 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.350834 (+ 2426us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.352068 (+ 1234us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.353677 (+ 1609us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.354159 (+ 482us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.357998 (+ 3839us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.358039 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.358045 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.358083 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.361924 (+ 3841us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.366021 (+ 4097us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.366027 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.366031 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.366154 (+ 123us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.366546 (+ 392us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.366552 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.366555 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.366869 (+ 314us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.367114 (+ 245us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.368095 (+ 981us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.368371 (+ 276us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":53}
I20251027 14:57:50.370214 24306 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:57:50.374048 24208 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } attempt: 5
Oct 27 14:57:50 dist-test-slave-hxxt krb5kdc[24049](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1761577069, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.104.61@KRBTEST.COM for kudu/127.0.104.60@KRBTEST.COM
I20251027 14:57:50.390699 24295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebbf2e21a4a4fb5a8ac62f7889b6e97. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.390913 24295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.385491 24065 master_runner.cc:186] Error getting master registration for 127.0.104.61:40075: OK, instance_id { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" instance_seqno: 1761577069655705 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20251027 14:57:50.392688 24065 master_runner.cc:418] Couldn't verify the masters in the cluster. Trying again...
I20251027 14:57:50.393695 24212 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.374587 (+ 0us) reactor.cc:625] Submitting negotiation task for client connection to 127.0.104.60:42789 (local address 127.0.0.1:59170)
1027 14:57:50.374949 (+ 362us) negotiation.cc:107] Waiting for socket to connect
1027 14:57:50.374969 (+ 20us) client_negotiation.cc:174] Beginning negotiation
1027 14:57:50.375082 (+ 113us) client_negotiation.cc:252] Sending NEGOTIATE NegotiatePB request
1027 14:57:50.376701 (+ 1619us) client_negotiation.cc:271] Received NEGOTIATE NegotiatePB response
1027 14:57:50.376707 (+ 6us) client_negotiation.cc:394] Received NEGOTIATE response from server
1027 14:57:50.377250 (+ 543us) client_negotiation.cc:189] Negotiated authn=SASL
1027 14:57:50.377767 (+ 517us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.377780 (+ 13us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.379681 (+ 1901us) client_negotiation.cc:271] Received TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.379687 (+ 6us) client_negotiation.cc:527] Received TLS_HANDSHAKE response from server
1027 14:57:50.380794 (+ 1107us) client_negotiation.cc:514] Sending TLS_HANDSHAKE message to server
1027 14:57:50.380808 (+ 14us) client_negotiation.cc:252] Sending TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.380943 (+ 135us) client_negotiation.cc:548] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.382331 (+ 1388us) client_negotiation.cc:623] Initiating SASL GSSAPI handshake
1027 14:57:50.382384 (+ 53us) client_negotiation.cc:656] calling sasl_client_start()
1027 14:57:50.386401 (+ 4017us) client_negotiation.cc:252] Sending SASL_INITIATE NegotiatePB request
1027 14:57:50.390676 (+ 4275us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.390681 (+ 5us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.390686 (+ 5us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.391132 (+ 446us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.391830 (+ 698us) client_negotiation.cc:271] Received SASL_CHALLENGE NegotiatePB response
1027 14:57:50.391836 (+ 6us) client_negotiation.cc:694] Received SASL_CHALLENGE response from server
1027 14:57:50.391840 (+ 4us) client_negotiation.cc:760] calling sasl_client_step()
1027 14:57:50.391955 (+ 115us) client_negotiation.cc:252] Sending SASL_RESPONSE NegotiatePB request
1027 14:57:50.392682 (+ 727us) client_negotiation.cc:271] Received SASL_SUCCESS NegotiatePB response
1027 14:57:50.392687 (+ 5us) client_negotiation.cc:715] Received SASL_SUCCESS response from server
1027 14:57:50.392836 (+ 149us) client_negotiation.cc:769] Sending connection context
1027 14:57:50.393020 (+ 184us) client_negotiation.cc:240] Negotiation successful
1027 14:57:50.393348 (+ 328us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"client-negotiator.queue_time_us":237}
I20251027 14:57:50.393738 24291 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.375263 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59170 (local address 127.0.104.60:42789)
1027 14:57:50.375428 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.375434 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.375452 (+ 18us) server_negotiation.cc:408] Connection header received
1027 14:57:50.375508 (+ 56us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.375515 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.375588 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.375676 (+ 88us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.378027 (+ 2351us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.379370 (+ 1343us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.381347 (+ 1977us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.381834 (+ 487us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.386638 (+ 4804us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
1027 14:57:50.386677 (+ 39us) server_negotiation.cc:882] Received SASL_INITIATE request from client
1027 14:57:50.386683 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
1027 14:57:50.386721 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
1027 14:57:50.390413 (+ 3692us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.391607 (+ 1194us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.391613 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.391616 (+ 3us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.391697 (+ 81us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
1027 14:57:50.392074 (+ 377us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
1027 14:57:50.392079 (+ 5us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
1027 14:57:50.392083 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
1027 14:57:50.392448 (+ 365us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
1027 14:57:50.392616 (+ 168us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.393146 (+ 530us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.393363 (+ 217us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":59}
I20251027 14:57:50.400780 24327 mvcc.cc:204] Tried to move back new op lower bound from 7215419680213852160 to 7215419679974801408. Current Snapshot: MvccSnapshot[applied={T|T < 7215419680213852160}]
I20251027 14:57:50.410315 24208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: Bootstrap starting.
I20251027 14:57:50.415141 24208 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: Neither blocks nor log segments found. Creating new log.
I20251027 14:57:50.416960 24208 log.cc:826] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: Log is configured to *not* fsync() on all Append() calls
I20251027 14:57:50.419139 24306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.419435 24306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:50.420620 24295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.420848 24295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.422539 24311 catalog_manager.cc:1357] Generated new cluster ID: a73a94bec96043a19dd541d1dc711c34
I20251027 14:57:50.422657 24311 catalog_manager.cc:1505] Initializing Kudu internal certificate authority...
I20251027 14:57:50.424081 24208 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: No bootstrap required, opened a new log
I20251027 14:57:50.429504 24208 raft_consensus.cc:359] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.429980 24208 raft_consensus.cc:740] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 51975d0d4311461fb8c1c9d755f5cfe1, State: Initialized, Role: FOLLOWER
I20251027 14:57:50.430553 24208 consensus_queue.cc:260] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } }
I20251027 14:57:50.434654 24335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.435001 24335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.436049 24208 sys_catalog.cc:565] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: configured and running, proceeding with master startup.
I20251027 14:57:50.436136 24183 tablet_service.cc:1911] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: true dest_uuid: "51975d0d4311461fb8c1c9d755f5cfe1"
I20251027 14:57:50.436477 24182 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [term 0 FOLLOWER]: Advancing to term 1
I20251027 14:57:50.446491 24182 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [term 1 FOLLOWER]: Refusing update from remote peer 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20251027 14:57:50.450006 24294 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [LEADER]: Connected to new peer: Peer: permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20251027 14:57:50.450654 24183 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2ebbf2e21a4a4fb5a8ac62f7889b6e97 in term 1.
I20251027 14:57:50.465536 24311 catalog_manager.cc:1380] Generated new certificate authority record
I20251027 14:57:50.467233 24311 catalog_manager.cc:1514] Loading token signing keys...
I20251027 14:57:50.477432 24348 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.477658 24335 sys_catalog.cc:455] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2ebbf2e21a4a4fb5a8ac62f7889b6e97. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.477784 24348 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.477820 24335 sys_catalog.cc:458] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1 [sys.catalog]: This master's current role is: FOLLOWER
I20251027 14:57:50.479179 24306 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2ebbf2e21a4a4fb5a8ac62f7889b6e97" member_type: VOTER last_known_addr { host: "127.0.104.62" port: 36773 } } peers { permanent_uuid: "51975d0d4311461fb8c1c9d755f5cfe1" member_type: VOTER last_known_addr { host: "127.0.104.61" port: 40075 } } peers { permanent_uuid: "241c5a1e86c64b94ba365f2f11a8482d" member_type: VOTER last_known_addr { host: "127.0.104.60" port: 42789 } } }
I20251027 14:57:50.479452 24306 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97 [sys.catalog]: This master's current role is: LEADER
I20251027 14:57:50.497278 24311 catalog_manager.cc:6022] T 00000000000000000000000000000000 P 2ebbf2e21a4a4fb5a8ac62f7889b6e97: Generated new TSK 0
I20251027 14:57:50.498497 24311 catalog_manager.cc:1524] Initializing in-progress tserver states...
I20251027 14:57:50.506887 24350 catalog_manager.cc:1269] Loaded cluster ID: a73a94bec96043a19dd541d1dc711c34
I20251027 14:57:50.507076 24350 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: loading cluster ID for follower catalog manager: success
I20251027 14:57:50.511194 24350 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: acquiring CA information for follower catalog manager: success
I20251027 14:57:50.513916 24350 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 51975d0d4311461fb8c1c9d755f5cfe1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20251027 14:57:50.593600 24290 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.578499 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:53378 (local address 127.0.104.62:36773)
1027 14:57:50.578691 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.578698 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.590937 (+ 12239us) server_negotiation.cc:408] Connection header received
1027 14:57:50.591511 (+ 574us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.591519 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.591609 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.591732 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.593344 (+ 1612us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53378: BlockingRecv error: recv got EOF from 127.0.0.1:53378 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
W20251027 14:57:50.596169 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.579130 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59186 (local address 127.0.104.60:42789)
1027 14:57:50.579578 (+ 448us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.579584 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.594546 (+ 14962us) server_negotiation.cc:408] Connection header received
1027 14:57:50.594810 (+ 264us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.594818 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.594921 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.594994 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.595933 (+ 939us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59186: BlockingRecv error: recv got EOF from 127.0.0.1:59186 (error 108)
Metrics: {"server-negotiator.queue_time_us":331}
W20251027 14:57:50.598423 24213 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.579392 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38574 (local address 127.0.104.61:40075)
1027 14:57:50.579550 (+ 158us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.579556 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.596870 (+ 17314us) server_negotiation.cc:408] Connection header received
1027 14:57:50.597108 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.597114 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.597198 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.597269 (+ 71us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.598198 (+ 929us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38574: BlockingRecv error: recv got EOF from 127.0.0.1:38574 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251027 14:57:50.611860 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.604206 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38578 (local address 127.0.104.61:40075)
1027 14:57:50.604419 (+ 213us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.604425 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.604892 (+ 467us) server_negotiation.cc:408] Connection header received
1027 14:57:50.605130 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.605137 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.605364 (+ 227us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.605537 (+ 173us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.606271 (+ 734us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.607742 (+ 1471us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.609534 (+ 1792us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.610136 (+ 602us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.610346 (+ 210us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.611130 (+ 784us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.611291 (+ 161us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.611413 (+ 122us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.611525 (+ 112us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20251027 14:57:50.627049 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.619181 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59192 (local address 127.0.104.60:42789)
1027 14:57:50.619355 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.619361 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.625322 (+ 5961us) server_negotiation.cc:408] Connection header received
1027 14:57:50.625650 (+ 328us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.625657 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.625808 (+ 151us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.625924 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.626793 (+ 869us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59192: BlockingRecv error: recv got EOF from 127.0.0.1:59192 (error 108)
Metrics: {"server-negotiator.queue_time_us":64}
I20251027 14:57:50.641280 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.634634 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38584 (local address 127.0.104.61:40075)
1027 14:57:50.634808 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.634814 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.635287 (+ 473us) server_negotiation.cc:408] Connection header received
1027 14:57:50.635460 (+ 173us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.635466 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.635569 (+ 103us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.635714 (+ 145us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.636458 (+ 744us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.637982 (+ 1524us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.639706 (+ 1724us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.640287 (+ 581us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.640366 (+ 79us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.640601 (+ 235us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.640709 (+ 108us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.640887 (+ 178us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.640996 (+ 109us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20251027 14:57:50.652614 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.650298 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59194 (local address 127.0.104.60:42789)
1027 14:57:50.650472 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.650478 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.650978 (+ 500us) server_negotiation.cc:408] Connection header received
1027 14:57:50.651248 (+ 270us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.651256 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.651341 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.651406 (+ 65us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.652354 (+ 948us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59194: BlockingRecv error: recv got EOF from 127.0.0.1:59194 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:57:50.665612 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.658695 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38596 (local address 127.0.104.61:40075)
1027 14:57:50.658874 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.658880 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.659337 (+ 457us) server_negotiation.cc:408] Connection header received
1027 14:57:50.659531 (+ 194us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.659538 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.659633 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.659742 (+ 109us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.660594 (+ 852us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.662103 (+ 1509us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.663828 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.664417 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.664758 (+ 341us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.664908 (+ 150us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.664969 (+ 61us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.665184 (+ 215us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.665291 (+ 107us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20251027 14:57:50.700268 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.697966 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59208 (local address 127.0.104.60:42789)
1027 14:57:50.698151 (+ 185us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.698157 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.698678 (+ 521us) server_negotiation.cc:408] Connection header received
1027 14:57:50.698926 (+ 248us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.698933 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.699040 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.699156 (+ 116us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.700013 (+ 857us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59208: BlockingRecv error: recv got EOF from 127.0.0.1:59208 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20251027 14:57:50.714900 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.708618 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38612 (local address 127.0.104.61:40075)
1027 14:57:50.708845 (+ 227us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.708852 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.709196 (+ 344us) server_negotiation.cc:408] Connection header received
1027 14:57:50.709394 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.709401 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.709494 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.709619 (+ 125us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.710331 (+ 712us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.711783 (+ 1452us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.713508 (+ 1725us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.714041 (+ 533us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.714120 (+ 79us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.714292 (+ 172us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.714389 (+ 97us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.714526 (+ 137us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.714626 (+ 100us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":94}
W20251027 14:57:50.723925 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.721486 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59218 (local address 127.0.104.60:42789)
1027 14:57:50.721762 (+ 276us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.721769 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.722529 (+ 760us) server_negotiation.cc:408] Connection header received
1027 14:57:50.722718 (+ 189us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.722724 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.722814 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.722914 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.723713 (+ 799us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59218: BlockingRecv error: recv got EOF from 127.0.0.1:59218 (error 108)
Metrics: {"server-negotiator.queue_time_us":162}
I20251027 14:57:50.741437 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.734938 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38624 (local address 127.0.104.61:40075)
1027 14:57:50.735134 (+ 196us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.735141 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.735614 (+ 473us) server_negotiation.cc:408] Connection header received
1027 14:57:50.735825 (+ 211us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.735831 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.735923 (+ 92us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.736014 (+ 91us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.736880 (+ 866us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.738275 (+ 1395us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.739939 (+ 1664us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.740530 (+ 591us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.740628 (+ 98us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.740784 (+ 156us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.740878 (+ 94us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.741049 (+ 171us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.741153 (+ 104us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":70}
W20251027 14:57:50.749936 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.747717 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59234 (local address 127.0.104.60:42789)
1027 14:57:50.747882 (+ 165us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.747887 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.748424 (+ 537us) server_negotiation.cc:408] Connection header received
1027 14:57:50.748608 (+ 184us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.748615 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.748721 (+ 106us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.748830 (+ 109us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.749678 (+ 848us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59234: BlockingRecv error: recv got EOF from 127.0.0.1:59234 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
I20251027 14:57:50.777384 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.770783 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:38626 (local address 127.0.104.61:40075)
1027 14:57:50.770967 (+ 184us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.770973 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.771462 (+ 489us) server_negotiation.cc:408] Connection header received
1027 14:57:50.771677 (+ 215us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.771683 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.771770 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.771865 (+ 95us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.772804 (+ 939us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.774225 (+ 1421us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.775895 (+ 1670us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.776435 (+ 540us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.776519 (+ 84us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.776716 (+ 197us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.776816 (+ 100us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.777000 (+ 184us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.777114 (+ 114us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20251027 14:57:50.788467 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.786279 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59244 (local address 127.0.104.60:42789)
1027 14:57:50.786458 (+ 179us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.786464 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.786983 (+ 519us) server_negotiation.cc:408] Connection header received
1027 14:57:50.787169 (+ 186us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.787176 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.787275 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.787388 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.788214 (+ 826us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59244: BlockingRecv error: recv got EOF from 127.0.0.1:59244 (error 108)
Metrics: {"server-negotiator.queue_time_us":69}
I20251027 14:57:50.831988 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.825301 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36064 (local address 127.0.104.61:40075)
1027 14:57:50.825475 (+ 174us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.825481 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.826068 (+ 587us) server_negotiation.cc:408] Connection header received
1027 14:57:50.826261 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.826267 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.826360 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.826475 (+ 115us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.827220 (+ 745us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.828768 (+ 1548us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.830471 (+ 1703us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.831022 (+ 551us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.831105 (+ 83us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.831272 (+ 167us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.831378 (+ 106us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.831539 (+ 161us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.831670 (+ 131us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":64}
W20251027 14:57:50.841250 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.838952 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39852 (local address 127.0.104.60:42789)
1027 14:57:50.839154 (+ 202us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.839161 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.839747 (+ 586us) server_negotiation.cc:408] Connection header received
1027 14:57:50.839921 (+ 174us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.839927 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.840016 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.840117 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.841011 (+ 894us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39852: BlockingRecv error: recv got EOF from 127.0.0.1:39852 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20251027 14:57:50.914613 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:50.908864 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36074 (local address 127.0.104.61:40075)
1027 14:57:50.909107 (+ 243us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.909114 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.909412 (+ 298us) server_negotiation.cc:408] Connection header received
1027 14:57:50.909664 (+ 252us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.909670 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.909766 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.909868 (+ 102us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:50.910691 (+ 823us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.911689 (+ 998us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:50.913398 (+ 1709us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:50.913791 (+ 393us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:50.913873 (+ 82us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:50.913994 (+ 121us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:50.914072 (+ 78us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:50.914227 (+ 155us) server_negotiation.cc:300] Negotiation successful
1027 14:57:50.914313 (+ 86us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":72}
W20251027 14:57:50.922566 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:50.920737 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39858 (local address 127.0.104.60:42789)
1027 14:57:50.920897 (+ 160us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:50.920902 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:50.921256 (+ 354us) server_negotiation.cc:408] Connection header received
1027 14:57:50.921417 (+ 161us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:50.921423 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:50.921522 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:50.921622 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:50.922333 (+ 711us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39858: BlockingRecv error: recv got EOF from 127.0.0.1:39858 (error 108)
Metrics: {"server-negotiator.queue_time_us":41}
I20251027 14:57:51.060674 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.055948 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36078 (local address 127.0.104.61:40075)
1027 14:57:51.056169 (+ 221us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.056176 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.056469 (+ 293us) server_negotiation.cc:408] Connection header received
1027 14:57:51.056694 (+ 225us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.056699 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.056767 (+ 68us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.056844 (+ 77us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.057463 (+ 619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.058365 (+ 902us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.059537 (+ 1172us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.059918 (+ 381us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.059976 (+ 58us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.060087 (+ 111us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.060166 (+ 79us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.060300 (+ 134us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.060421 (+ 121us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":77}
W20251027 14:57:51.067162 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:51.065660 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39868 (local address 127.0.104.60:42789)
1027 14:57:51.065798 (+ 138us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.065802 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.066130 (+ 328us) server_negotiation.cc:408] Connection header received
1027 14:57:51.066264 (+ 134us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.066269 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.066338 (+ 69us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.066417 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:51.066979 (+ 562us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39868: BlockingRecv error: recv got EOF from 127.0.0.1:39868 (error 108)
Metrics: {"server-negotiator.queue_time_us":45}
I20251027 14:57:51.331113 24213 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.326640 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36080 (local address 127.0.104.61:40075)
1027 14:57:51.326833 (+ 193us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.326840 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.327008 (+ 168us) server_negotiation.cc:408] Connection header received
1027 14:57:51.327171 (+ 163us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.327176 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.327248 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.327330 (+ 82us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.327933 (+ 603us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.328821 (+ 888us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.330075 (+ 1254us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.330395 (+ 320us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.330451 (+ 56us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.330563 (+ 112us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.330632 (+ 69us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.330757 (+ 125us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.330835 (+ 78us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
W20251027 14:57:51.338496 24291 negotiation.cc:337] Failed RPC negotiation. Trace:
1027 14:57:51.336523 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39874 (local address 127.0.104.60:42789)
1027 14:57:51.336715 (+ 192us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.336720 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.336963 (+ 243us) server_negotiation.cc:408] Connection header received
1027 14:57:51.337166 (+ 203us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.337172 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.337262 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.337362 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
1027 14:57:51.338255 (+ 893us) negotiation.cc:327] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39874: BlockingRecv error: recv got EOF from 127.0.0.1:39874 (error 108)
Metrics: {"server-negotiator.queue_time_us":47}
I20251027 14:57:51.339823 24320 catalog_manager.cc:1269] Loaded cluster ID: a73a94bec96043a19dd541d1dc711c34
I20251027 14:57:51.339916 24320 catalog_manager.cc:1562] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: loading cluster ID for follower catalog manager: success
I20251027 14:57:51.342582 24320 catalog_manager.cc:1584] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: acquiring CA information for follower catalog manager: success
I20251027 14:57:51.344449 24320 catalog_manager.cc:1612] T 00000000000000000000000000000000 P 241c5a1e86c64b94ba365f2f11a8482d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20251027 14:57:51.860574 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.855644 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36084 (local address 127.0.104.61:40075)
1027 14:57:51.856106 (+ 462us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.856112 (+ 6us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.856177 (+ 65us) server_negotiation.cc:408] Connection header received
1027 14:57:51.856420 (+ 243us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.856425 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.856504 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.856602 (+ 98us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.857289 (+ 687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.858211 (+ 922us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.859403 (+ 1192us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.859778 (+ 375us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.859848 (+ 70us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.859963 (+ 115us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.860039 (+ 76us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.860180 (+ 141us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.860276 (+ 96us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":357,"thread_start_us":249,"threads_started":1}
I20251027 14:57:51.871482 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.866255 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39880 (local address 127.0.104.60:42789)
1027 14:57:51.866620 (+ 365us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.866625 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.866736 (+ 111us) server_negotiation.cc:408] Connection header received
1027 14:57:51.866926 (+ 190us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.866930 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.867067 (+ 137us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.867193 (+ 126us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.867809 (+ 616us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.868748 (+ 939us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.869911 (+ 1163us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.870220 (+ 309us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.870372 (+ 152us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.870912 (+ 540us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.871028 (+ 116us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.871170 (+ 142us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.871258 (+ 88us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":276,"thread_start_us":196,"threads_started":1}
I20251027 14:57:51.886807 24481 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.877667 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:59610 (local address 127.0.104.62:36773)
1027 14:57:51.880736 (+ 3069us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.880740 (+ 4us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.880756 (+ 16us) server_negotiation.cc:408] Connection header received
1027 14:57:51.880861 (+ 105us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.880866 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.881002 (+ 136us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.881150 (+ 148us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.882102 (+ 952us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.883440 (+ 1338us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.885059 (+ 1619us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.885446 (+ 387us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.885565 (+ 119us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.885898 (+ 333us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.886001 (+ 103us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.886418 (+ 417us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.886515 (+ 97us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":2980,"thread_start_us":2890,"threads_started":1}
I20251027 14:57:51.887159 24474 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.877518 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:39894 (local address 127.0.104.60:42789)
1027 14:57:51.881712 (+ 4194us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.881719 (+ 7us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.882281 (+ 562us) server_negotiation.cc:408] Connection header received
1027 14:57:51.882433 (+ 152us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.882437 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.882514 (+ 77us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.882588 (+ 74us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.883438 (+ 850us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.884344 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.885983 (+ 1639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.886417 (+ 434us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.886484 (+ 67us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.886614 (+ 130us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.886704 (+ 90us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.886843 (+ 139us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.886949 (+ 106us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":4070}
I20251027 14:57:51.888248 24468 negotiation.cc:339] RPC negotiation tracing enabled. Trace:
1027 14:57:51.877450 (+ 0us) reactor.cc:625] Submitting negotiation task for server connection from 127.0.0.1:36094 (local address 127.0.104.61:40075)
1027 14:57:51.877621 (+ 171us) server_negotiation.cc:207] Beginning negotiation
1027 14:57:51.877626 (+ 5us) server_negotiation.cc:400] Waiting for connection header
1027 14:57:51.881822 (+ 4196us) server_negotiation.cc:408] Connection header received
1027 14:57:51.882088 (+ 266us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
1027 14:57:51.882093 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
1027 14:57:51.882180 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
1027 14:57:51.882246 (+ 66us) server_negotiation.cc:227] Negotiated authn=TOKEN
1027 14:57:51.883335 (+ 1089us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.884641 (+ 1306us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
1027 14:57:51.886318 (+ 1677us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
1027 14:57:51.886716 (+ 398us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
1027 14:57:51.886787 (+ 71us) server_negotiation.cc:366] Received TOKEN_EXCHANGE NegotiatePB request
1027 14:57:51.886936 (+ 149us) server_negotiation.cc:378] Sending TOKEN_EXCHANGE NegotiatePB response
1027 14:57:51.887017 (+ 81us) server_negotiation.cc:1036] Waiting for connection context
1027 14:57:51.887904 (+ 887us) server_negotiation.cc:300] Negotiation successful
1027 14:57:51.888007 (+ 103us) negotiation.cc:327] Negotiation complete: OK
Metrics: {"server-negotiator.queue_time_us":65}
I20251027 14:57:51.891741 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 24065
I20251027 14:57:51.892926 24127 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:52.000877 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 24065
I20251027 14:57:52.018276 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 24141
I20251027 14:57:52.019340 24203 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:52.126279 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 24141
I20251027 14:57:52.142987 416 external_mini_cluster.cc:1699] Attempting to check leaks for /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu pid 24218
I20251027 14:57:52.144050 24280 generic_service.cc:196] Checking for leaks (request via RPC)
I20251027 14:57:52.249691 416 external_mini_cluster.cc:1658] Killing /tmp/dist-test-taskuw8ijJ/build/asan/bin/kudu with pid 24218
2025-10-27T14:57:52Z chronyd exiting
[ OK ] ConnectToFollowerMasterTest.AuthnTokenVerifierHaveKeys/1 (5740 ms)
[----------] 2 tests from ConnectToFollowerMasterTest (11151 ms total)
[----------] Global test environment tear-down
[==========] 57 tests from 4 test suites ran. (438483 ms total)
[ PASSED ] 56 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] AuthTokenIssuingTest.ChannelConfidentiality/3, where GetParam() = 80-byte object <02-00 00-00 BE-BE BE-BE E8-24 01-00 70-60 00-00 08-00 00-00 00-00 00-00 72-65 71-75 69-72 65-64 00-BE BE-BE BE-BE BE-BE 08-25 01-00 70-60 00-00 08-00 00-00 00-00 00-00 72-65 71-75 69-72 65-64 00-BE BE-BE BE-BE BE-BE 00-00 01-BE BE-BE BE-BE>
1 FAILED TEST